Add encryption, Chromium tests and a README
CI / Node 24 (push) Successful in 15s
CI / Node 26 (push) Successful in 16s
CI / Node 22 (pull_request) Successful in 13s
CI / Node 24 (pull_request) Failing after 3s
CI / Node 26 (pull_request) Failing after 4s
CI / Chromium (push) Failing after 1m8s
CI / Publish to staging registry (push) Skipped
CI / Chromium (pull_request) Failing after 1m3s
CI / Publish to staging registry (pull_request) Skipped
CI / Node 22 (push) Failing after 4s
CI / Node 24 (push) Successful in 15s
CI / Node 26 (push) Successful in 16s
CI / Node 22 (pull_request) Successful in 13s
CI / Node 24 (pull_request) Failing after 3s
CI / Node 26 (pull_request) Failing after 4s
CI / Chromium (push) Failing after 1m8s
CI / Publish to staging registry (push) Skipped
CI / Chromium (pull_request) Failing after 1m3s
CI / Publish to staging registry (pull_request) Skipped
CI / Node 22 (push) Failing after 4s
- KVStorageEncrypted: AES-GCM per value (fresh IV, storage key as AAD), keys stay readable; decrypts and re-emits pushed changes and passes subscriptions through, so it can be a composite source. generateKey / exportKey / importKey / deriveKey (PBKDF2). - KVStorageHttpServer: allowOrigins (CORS), including cache-control for SSE preflights. - Chromium tests (Vitest browser mode + Playwright) against a real cross-origin server: conformance for IndexedDB, encrypted, HTTP and composite stores; signed requests and SSE; two devices syncing, offline writes surviving a reload. New CI job. - Composite fixes found by those tests: sync() during a running sync schedules one more (changes between the first scan and the stream connecting were missed); a queued write the source refuses is dropped, restored and reported (onError) instead of blocking the queue forever. - README with examples, mirrored in test/readme.test.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
This commit is contained in:
@@ -33,13 +33,16 @@ section 7 lists what changed and why.
|
||||
```
|
||||
src/
|
||||
kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts,
|
||||
kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts
|
||||
kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts,
|
||||
kv-storage-encrypted.ts
|
||||
event-storage/ base-event-storage.ts, event-storage-kv.ts
|
||||
document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
|
||||
event-sourced-document-storage.ts
|
||||
utils/ keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types),
|
||||
document-path/query/update, sse-session/ (vendored)
|
||||
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
|
||||
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts;
|
||||
browser/ runs in headless Chromium against a real cross-origin server;
|
||||
readme.test.ts runs the README's examples
|
||||
bench/
|
||||
```
|
||||
|
||||
@@ -99,7 +102,10 @@ no own value. A node's own value and its children are independent keys.
|
||||
`scan` streams NDJSON. Unreachable server (fetch failure, 502/503/504) →
|
||||
`StorageUnavailableError`. The server executes exactly what clients send and never reads values
|
||||
for decisions, so values can be encrypted end to end.
|
||||
- `KVStorageComposite`: see 4.2.
|
||||
- `KVStorageComposite`: see 4.2. `KVStorageEncrypted`: see 4.3.
|
||||
- **CORS.** `allowOrigins` (`['https://app…']` or `'*'`; default none) answers preflights and adds
|
||||
`Access-Control-Allow-Origin` for allowed origins. Allowed request headers: `content-type`,
|
||||
`cache-control` (sent by `SSESession`), `last-event-id` and the auth headers.
|
||||
- **Change stream (SSE).** `POST /events` with `{ paths }` streams every change to those keys and
|
||||
their subtrees as `event: change` (data: the changes), including the subscriber's own writes.
|
||||
`event: ready` follows each connection; keep-alive comments are sent every 15 s. The server keeps
|
||||
@@ -109,6 +115,8 @@ no own value. A node's own value and its children are independent keys.
|
||||
new list; `subscribe` resolves once connected. A 401/403 rejects instead of retrying.
|
||||
- The client emits each pushed batch as `change` (`origin: 'remote'`), and `resync` whenever the
|
||||
stream reconnects after a drop, since changes may have been missed meanwhile.
|
||||
- One SSE connection per `KVStorageHttp` instance. Browsers allow 6 HTTP/1.1 connections per
|
||||
host, so apps should share one client per server (or use HTTP/2); `close()` releases it.
|
||||
- Limitation: a server only sees writes made through its own storage instance, so several server
|
||||
processes over one database need a shared pub/sub (not built).
|
||||
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
|
||||
@@ -159,20 +167,41 @@ is only appropriate when the client may read all of it.
|
||||
- A write the source refuses (e.g. 403) is undone locally and rethrown.
|
||||
- On `StorageUnavailableError`, or while a sync runs or writes are queued (to keep order), the
|
||||
write goes to the queue and is sent later. The last write to reach the source wins.
|
||||
- A queued write the source refuses on replay is dropped (retrying would block every later
|
||||
write), the source's values for its keys are restored locally, and it is reported to `onError`.
|
||||
- **Remote changes** pushed by the source are applied and announced (`change`, `origin: 'remote'`)
|
||||
only where they differ from what the first replica holds. No write tagging or echo tracking is
|
||||
needed: echoes of this client's writes change nothing.
|
||||
- **`sync()`**: replay the queue in order (send, then re-apply locally) → merge-scan the source
|
||||
against the last replica (both key-ordered), applying the difference in batches of 500 →
|
||||
replay again (writes made meanwhile) → mark `synced`. It runs on start, on `resync` (the stream
|
||||
reconnected), and `retryMs` after a failure while offline.
|
||||
reconnected), and `retryMs` after a failure while offline. Called while a sync is running, it
|
||||
schedules one more after it (shared by all callers meanwhile), so changes after the running
|
||||
sync's scan aren't missed; e.g. the sync after subscribing, which closes the gap between the
|
||||
first scan and the stream connecting.
|
||||
- **Reads** wait for the first sync. Until one has completed, they go to the source. With a
|
||||
persistent queue, a restart reads locally straight away (offline too).
|
||||
- `start()` (also lazy on first use), `sync()`, `queued`, `close()`.
|
||||
- `start()` (also lazy on first use), `sync()`, `queued`, `close()`; `onError` receives errors no
|
||||
caller can be rejected with (refused queued writes, failed syncs). Default `console.error`.
|
||||
- Limitations: catching up re-scans the whole range (no server history); a remote change can
|
||||
overwrite a value whose write is still queued until that write is replayed (last write wins);
|
||||
replicas, source and queue must be root stores (the primitives use absolute keys).
|
||||
|
||||
### 4.3 `KVStorageEncrypted(inner, { key })`
|
||||
|
||||
Encrypts values before they reach another KV storage (usually `KVStorageHttp`), so the server only
|
||||
stores ciphertext.
|
||||
|
||||
- AES-GCM (WebCrypto) with a fresh 96-bit random IV per write. Stored value: `Uint8Array`
|
||||
`[1 (format), iv (12), ciphertext + tag]`. The storage key is the additional authenticated data,
|
||||
so a value moved to another key, or altered, fails to decrypt (`Error`, with the cause).
|
||||
- Keys stay plaintext (the server needs them for authorization, ordering and prefix scans); value
|
||||
sizes and timing are visible.
|
||||
- Remote `change` events from the inner storage are decrypted (in order) and re-emitted; `resync`
|
||||
and `subscribe` / `unsubscribe` pass through, so it works as a composite `source`.
|
||||
- Keys: `generateKey()` (extractable), `exportKey` / `importKey` (raw), `deriveKey(password, salt,
|
||||
iterations = 600 000)` (PBKDF2-SHA-256, not extractable). No key rotation yet.
|
||||
|
||||
## 5. Event storage
|
||||
|
||||
`BaseEventStorage` primitives over stream paths: `appendTo(stream, events)` → ids,
|
||||
@@ -258,6 +287,8 @@ Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)`
|
||||
re-scan (would need server-side history or a change log in event storage).
|
||||
- Event-sourced documents: subscribing to event streams to run `catchUp` automatically.
|
||||
- Authorization beyond "public key = user id" (sharing a subtree with other keys).
|
||||
- Encryption: key rotation (re-encrypt, or a key id in the value format); sharing encrypted data
|
||||
between users.
|
||||
- Multi-process servers need a shared change feed (pub/sub) for SSE.
|
||||
|
||||
- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The
|
||||
|
||||
Reference in New Issue
Block a user