Add encryption, Chromium tests and a README
CI / Node 24 (push) Successful in 15s
CI / Node 26 (push) Successful in 16s
CI / Node 22 (pull_request) Successful in 13s
CI / Node 24 (pull_request) Failing after 3s
CI / Node 26 (pull_request) Failing after 4s
CI / Chromium (push) Failing after 1m8s
CI / Publish to staging registry (push) Skipped
CI / Chromium (pull_request) Failing after 1m3s
CI / Publish to staging registry (pull_request) Skipped
CI / Node 22 (push) Failing after 4s

- KVStorageEncrypted: AES-GCM per value (fresh IV, storage key as AAD), keys stay
  readable; decrypts and re-emits pushed changes and passes subscriptions through, so it
  can be a composite source. generateKey / exportKey / importKey / deriveKey (PBKDF2).
- KVStorageHttpServer: allowOrigins (CORS), including cache-control for SSE preflights.
- Chromium tests (Vitest browser mode + Playwright) against a real cross-origin server:
  conformance for IndexedDB, encrypted, HTTP and composite stores; signed requests and
  SSE; two devices syncing, offline writes surviving a reload. New CI job.
- Composite fixes found by those tests: sync() during a running sync schedules one more
  (changes between the first scan and the stream connecting were missed); a queued write
  the source refuses is dropped, restored and reported (onError) instead of blocking the
  queue forever.
- README with examples, mirrored in test/readme.test.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
This commit is contained in:
Claude
2026-10-23 14:25:22 +00:00
parent 3525dbc622
commit 68d1a1c8e9
21 changed files with 1387 additions and 24 deletions
+36 -5
View File
@@ -33,13 +33,16 @@ section 7 lists what changed and why.
```
src/
kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts,
kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts
kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts,
kv-storage-encrypted.ts
event-storage/ base-event-storage.ts, event-storage-kv.ts
document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
event-sourced-document-storage.ts
utils/ keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types),
document-path/query/update, sse-session/ (vendored)
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts;
browser/ runs in headless Chromium against a real cross-origin server;
readme.test.ts runs the README's examples
bench/
```
@@ -99,7 +102,10 @@ no own value. A node's own value and its children are independent keys.
`scan` streams NDJSON. Unreachable server (fetch failure, 502/503/504) →
`StorageUnavailableError`. The server executes exactly what clients send and never reads values
for decisions, so values can be encrypted end to end.
- `KVStorageComposite`: see 4.2.
- `KVStorageComposite`: see 4.2. `KVStorageEncrypted`: see 4.3.
- **CORS.** `allowOrigins` (`['https://app…']` or `'*'`; default none) answers preflights and adds
`Access-Control-Allow-Origin` for allowed origins. Allowed request headers: `content-type`,
`cache-control` (sent by `SSESession`), `last-event-id` and the auth headers.
- **Change stream (SSE).** `POST /events` with `{ paths }` streams every change to those keys and
their subtrees as `event: change` (data: the changes), including the subscriber's own writes.
`event: ready` follows each connection; keep-alive comments are sent every 15 s. The server keeps
@@ -109,6 +115,8 @@ no own value. A node's own value and its children are independent keys.
new list; `subscribe` resolves once connected. A 401/403 rejects instead of retrying.
- The client emits each pushed batch as `change` (`origin: 'remote'`), and `resync` whenever the
stream reconnects after a drop, since changes may have been missed meanwhile.
- One SSE connection per `KVStorageHttp` instance. Browsers allow 6 HTTP/1.1 connections per
host, so apps should share one client per server (or use HTTP/2); `close()` releases it.
- Limitation: a server only sees writes made through its own storage instance, so several server
processes over one database need a shared pub/sub (not built).
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
@@ -159,20 +167,41 @@ is only appropriate when the client may read all of it.
- A write the source refuses (e.g. 403) is undone locally and rethrown.
- On `StorageUnavailableError`, or while a sync runs or writes are queued (to keep order), the
write goes to the queue and is sent later. The last write to reach the source wins.
- A queued write the source refuses on replay is dropped (retrying would block every later
write), the source's values for its keys are restored locally, and it is reported to `onError`.
- **Remote changes** pushed by the source are applied and announced (`change`, `origin: 'remote'`)
only where they differ from what the first replica holds. No write tagging or echo tracking is
needed: echoes of this client's writes change nothing.
- **`sync()`**: replay the queue in order (send, then re-apply locally) → merge-scan the source
against the last replica (both key-ordered), applying the difference in batches of 500 →
replay again (writes made meanwhile) → mark `synced`. It runs on start, on `resync` (the stream
reconnected), and `retryMs` after a failure while offline.
reconnected), and `retryMs` after a failure while offline. Called while a sync is running, it
schedules one more after it (shared by all callers meanwhile), so changes after the running
sync's scan aren't missed; e.g. the sync after subscribing, which closes the gap between the
first scan and the stream connecting.
- **Reads** wait for the first sync. Until one has completed, they go to the source. With a
persistent queue, a restart reads locally straight away (offline too).
- `start()` (also lazy on first use), `sync()`, `queued`, `close()`.
- `start()` (also lazy on first use), `sync()`, `queued`, `close()`; `onError` receives errors no
caller can be rejected with (refused queued writes, failed syncs). Default `console.error`.
- Limitations: catching up re-scans the whole range (no server history); a remote change can
overwrite a value whose write is still queued until that write is replayed (last write wins);
replicas, source and queue must be root stores (the primitives use absolute keys).
### 4.3 `KVStorageEncrypted(inner, { key })`
Encrypts values before they reach another KV storage (usually `KVStorageHttp`), so the server only
stores ciphertext.
- AES-GCM (WebCrypto) with a fresh 96-bit random IV per write. Stored value: `Uint8Array`
`[1 (format), iv (12), ciphertext + tag]`. The storage key is the additional authenticated data,
so a value moved to another key, or altered, fails to decrypt (`Error`, with the cause).
- Keys stay plaintext (the server needs them for authorization, ordering and prefix scans); value
sizes and timing are visible.
- Remote `change` events from the inner storage are decrypted (in order) and re-emitted; `resync`
and `subscribe` / `unsubscribe` pass through, so it works as a composite `source`.
- Keys: `generateKey()` (extractable), `exportKey` / `importKey` (raw), `deriveKey(password, salt,
iterations = 600 000)` (PBKDF2-SHA-256, not extractable). No key rotation yet.
## 5. Event storage
`BaseEventStorage` primitives over stream paths: `appendTo(stream, events)` → ids,
@@ -258,6 +287,8 @@ Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)`
re-scan (would need server-side history or a change log in event storage).
- Event-sourced documents: subscribing to event streams to run `catchUp` automatically.
- Authorization beyond "public key = user id" (sharing a subtree with other keys).
- Encryption: key rotation (re-encrypt, or a key id in the value format); sharing encrypted data
between users.
- Multi-process servers need a shared change feed (pub/sub) for SSE.
- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The