Authenticated, scoped sync: signed requests and per-path subscriptions
CI / Node 24 (pull_request) Successful in 12s
CI / Node 22 (pull_request) Successful in 13s
CI / Node 26 (pull_request) Successful in 13s

- RequestAuth: clients sign `${timestamp}:${path}:${body}` (as in
  sync-server-example); headers carry public key, timestamp, signature and
  algorithm. Pluggable signers/verifiers; Ed25519 via WebCrypto built in.
  Timestamp window and replay protection.
- KVStorageHttpServer requires auth ('none' must be explicit). With auth, every
  key, scan range and subscription path must be under the caller's public key.
- POST /events with { paths }: subscribers only receive (and on resume only
  replay) changes under their authorized paths.
- KVStorageHttp: signer option; subscribe(path)/unsubscribe(path)/close()
  replace watch(); 401/403 reject instead of retrying.
- KVStorageComposite: prefix option scopes sync, cache and subscription.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
This commit is contained in:
Claude
2026-10-23 10:01:06 +00:00
parent df7de27e20
commit a8c9539ad9
14 changed files with 696 additions and 129 deletions
+43 -10
View File
@@ -94,26 +94,58 @@ Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `
`handle(Request) → Response`): `POST /<primitive>` with Extended JSON bodies; `scan` streams
NDJSON. Unreachable server (fetch failure, 502/503/504) → `StorageUnavailableError`. The server
exposes the whole underlying store (primitives use absolute keys).
- `KVStorageComposite`: see 4.1.
- **Change stream (SSE).** `GET /events` on the server streams every change to the wrapped storage
as `event: change` with id `<epoch>-<seq>` (epoch = server instance). The server keeps the last
- `KVStorageComposite`: see 4.2.
- **Change stream (SSE).** `POST /events` with `{ paths }` streams changes to those keys and their
subtrees as `event: change` with id `<epoch>-<seq>` (epoch = server instance). The server keeps the last
`historySize` (10 000) change batches.
- A client reconnecting with `Last-Event-ID` gets the batches it missed, or `event: resync` if it
can't be resumed (another epoch, or too old).
- Every catch-up ends with `event: ready`.
- Keep-alive comments are sent every 15 s.
- `KVStorageHttp.watch()` subscribes (via the vendored `SSESession`, which reconnects with
backoff), emits `change` events with `origin: 'remote'` and `resync`, and resolves once caught up.
- `KVStorageHttp.subscribe(path = this.key)` / `unsubscribe(path)` manage the subscribed paths.
Each change reconnects the stream (vendored `SSESession`, backoff) with the new list, resuming
from `lastEventId`. The client emits `change` (`origin: 'remote'`) and `resync`, and `subscribe`
resolves once caught up. A 401/403 rejects `subscribe` instead of retrying.
- Limitation: a server only sees writes made through its own storage instance, so several server
processes over one database need a shared pub/sub (not built).
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be
encrypted freely.
### 4.1 `KVStorageComposite({ source, replicas, retry? })`
### 4.1 Authentication (`utils/request-auth.ts`)
Keeps full local copies of a source (e.g. `KVStorageHttp`) so reads are local and the app works
offline. Replicas are listed fastest first (e.g. `[inMemory, indexedDb]`). The last replica is the
The server requires `auth`: either `{ verifiers, maxSkewMs?, now? }`, or `'none'` (open, for local
development and tests). It can't be left open by omission.
- **Signed requests.** The client's `RequestSigner` signs `${timestamp}:${path}:${body}` (path =
URL pathname, body = the exact request body; same scheme as Harvmaster/sync-server-example).
Headers: `X-Public-Key` (lowercase hex), `X-Timestamp` (ms), `X-Signature` (hex) and
`X-Signature-Algorithm`. Every request is signed, and so is each SSE (re)connect, with a fresh
timestamp.
- **Verification (401).** The server picks the `SignatureVerifier` registered for the algorithm
(unknown → 401) and checks the timestamp is within `maxSkewMs` (default 60 s). It then rejects a
signature already accepted within that window (replay protection), and verifies the signature
over the received path and raw body. The body is parsed only afterwards.
- **Algorithms are pluggable.** Built in: `Ed25519Signer` / `Ed25519Verifier` (WebCrypto, so no
dependencies; Node 22+ and current browsers; keys persist as PKCS #8). Others, e.g. secp256k1 via
libauth, implement `RequestSigner` (`algorithm`, `publicKey`, `sign`) and `SignatureVerifier`
(`algorithm`, `verify`). The server can accept several algorithms at once.
- **Authorization (403): the public key is the user id.** Every key in `readMany`, `writeMany`,
`removeMany` and `writeIfAbsent`, and every subscription path, must be `<publicKey>` or under
`<publicKey>/`. A scan must stay within `[<publicKey>, <publicKey>0)`, i.e. the key and its
subtree (escaping guarantees no other key falls in that range, including lookalikes such as
`<publicKey>x`).
- **Scoped events.** Each subscriber only receives changes under its own (authorized) paths, live
and when resuming. Event ids are a global sequence, so a client can infer the server's overall
write volume, though nothing about the contents.
- Proxies must preserve the URL path, because the signature covers it.
### 4.2 `KVStorageComposite({ source, prefix?, replicas, retry? })`
Keeps local copies of a source (e.g. `KVStorageHttp`) so reads are local and the app works
offline. With `prefix` (e.g. the user's public key) only that key and its subtree are synced, cached
and subscribed to; keys outside it throw `RangeError`. Without it, the whole source is copied, which
is only appropriate when the client may read all of it. Replicas are listed fastest first (e.g. `[inMemory, indexedDb]`). The last replica is the
persistent one: it seeds the others on start and stores the composite's internal keys under `~q/`
(hidden from reads): `~q/meta` (`{ lastEventId }`, present once a sync has completed) and the
outbox `~q/o/<seq:12>`.
@@ -222,8 +254,9 @@ Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)`
## 8. Open questions
- Composite: replicate a subtree instead of the whole store; conflict policies beyond last writer
wins for offline writes.
- Composite: lazily syncing parts of a large subtree on first read; conflict policies beyond last
writer wins for offline writes.
- Authorization beyond "public key = user id" (sharing a subtree with other keys).
- Multi-process servers need a shared change feed (pub/sub) for SSE.
- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The