Add KVStorageComposite, IndexedDB storage and SSE change streams
- Every KV storage emits change events (base class owns writeMany/removeMany/ writeIfAbsent; implementations provide protected apply* methods) - KVStorageHttpServer streams changes over SSE (GET /events) with a resumable history; KVStorageHttp.watch() subscribes and resumes via Last-Event-ID - KVStorageIndexedDB for local persistence - KVStorageComposite keeps replicas of a source in sync: local reads, resume instead of full sync, offline outbox replayed in order, echo tracking - Values may contain bigint/Uint8Array (Extended JSON over the network) - Vendor sse-session, event-emitter and extended-json (no runtime deps) - Freeze child nodes so shadowed state on children fails loudly - CI on Node 22, 24 and 26 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
This commit is contained in:
Claude
@@ -14,6 +14,9 @@ section 7 lists what changed and why.
|
||||
that delegates the primitives.
|
||||
2. **One navigation primitive: `child(key)`.** Every instance is a node; `child` returns a view of
|
||||
the same storage rooted deeper (via `Object.create`, so it shares the instance's state). Sync, no I/O.
|
||||
Child nodes are **frozen**: an implementation that reassigns a field while running on a child
|
||||
would silently shadow the root's value, so that throws instead. Implementations keep mutable
|
||||
state in a `readonly state` object or in Maps.
|
||||
3. **Self or child by argument.** `get()` reads this node, `get(key)` a child. A key that is passed
|
||||
but `undefined`/`null` throws (`InvalidKeyError`) instead of silently meaning "self". Methods use
|
||||
optional parameters or tuple-union rest parameters, not overloads.
|
||||
@@ -21,16 +24,21 @@ section 7 lists what changed and why.
|
||||
suits the backend (in-memory equality maps, native Mongo/SQLite/IndexedDB indexes, or nothing).
|
||||
5. **Simple over clever.** Code style: classes, TS `private` (not `#`), no method overrides in
|
||||
subclasses, tests only under `test/`.
|
||||
6. **No runtime dependencies.** Third-party code is vendored into `src/utils/` with a header naming
|
||||
its source: `sse-session/` (Harvmaster/sse-session), `event-emitter.ts` and `extended-json.ts`
|
||||
(xo-cash-utils, branch installable-sse). Node ≥ 22; CI runs Node 22, 24 and 26.
|
||||
|
||||
## 2. Layout
|
||||
|
||||
```
|
||||
src/
|
||||
kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-http.ts, kv-storage-http-server.ts
|
||||
kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts,
|
||||
kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts
|
||||
event-storage/ base-event-storage.ts, event-storage-kv.ts
|
||||
document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
|
||||
event-sourced-document-storage.ts
|
||||
utils/ keys, sorted-map, json, ulid, errors, schema (types), document-path/query/update
|
||||
utils/ keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types),
|
||||
document-path/query/update, sse-session/ (vendored)
|
||||
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
|
||||
bench/
|
||||
```
|
||||
@@ -49,15 +57,24 @@ bench/
|
||||
|
||||
## 4. KV storage
|
||||
|
||||
`BaseKVStorage` primitives (absolute keys):
|
||||
`BaseKVStorage` primitives (absolute keys). Implementations provide the reads and the protected
|
||||
`apply*` writes; the public `writeMany` / `removeMany` / `writeIfAbsent` live in the base class and
|
||||
emit a `change` event after each write, so every store emits change events.
|
||||
|
||||
| Primitive | |
|
||||
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| `readMany(keys)` | values in order, `undefined` for missing |
|
||||
| `writeMany(entries)` | |
|
||||
| `removeMany(keys)` | |
|
||||
| `writeIfAbsent(entries)` | `true` = written; atomic per key (needed by events) |
|
||||
| `scan({ gte, lt, limit, reverse, values })` | key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once) |
|
||||
| Primitive | |
|
||||
| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
|
||||
| `readMany(keys)` | values in order, `undefined` for missing |
|
||||
| `scan({ gte, lt, limit, reverse, values })` | key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once) |
|
||||
| `applyWrite(entries)` → public `writeMany` | |
|
||||
| `applyRemove(keys)` → public `removeMany` | |
|
||||
| `applyWriteIfAbsent(entries)` → public `writeIfAbsent` | `true` = written; atomic per key (needed by events) |
|
||||
|
||||
**Values** are JSON plus `bigint` and `Uint8Array`. Anything crossing the network is encoded as
|
||||
Extended JSON (`<bigint: 123n>`, `<uint8array: abcd>`); IndexedDB stores them natively.
|
||||
|
||||
**Events** (`EventEmitter`, shared by all nodes of a storage): `change` →
|
||||
`{ changes: { key, value? }[], origin: 'local' | 'remote', id? }` (no `value` = removed), and
|
||||
`resync` → changes may have been missed, so reload.
|
||||
|
||||
Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `set(key, value)`,
|
||||
`setMany`, `delete(key?)`, `deleteMany`, `deleteTree(key?)`, `keys({ limit, after, reverse })`,
|
||||
@@ -68,14 +85,66 @@ Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `
|
||||
reserved segments.
|
||||
- **The only KV "index" is the ordered key space.** There is no `find`: with KV you know where to look,
|
||||
or you list a node's children.
|
||||
- **Implementations:** `KVStorageInMemory` (`SortedMap`: sorted chunks of ≤1024 keys; values deep-copied);
|
||||
`KVStorageHttp` (client) + `KVStorageHttpServer` (wraps any KV storage; `handle(Request) → Response`,
|
||||
`POST /<primitive>` with JSON bodies, `scan` streamed as NDJSON). The server exposes the whole
|
||||
underlying store (primitives use absolute keys).
|
||||
- **Implementations:**
|
||||
- `KVStorageInMemory`: `SortedMap` (sorted chunks of ≤1024 keys); values deep-copied.
|
||||
- `KVStorageIndexedDB`: one object store with string keys. Scans read in batches, each in its own
|
||||
transaction, resuming after the last key (IndexedDB transactions close when the code awaits
|
||||
anything else). `writeIfAbsent` runs in one readwrite transaction.
|
||||
- `KVStorageHttp` (client) and `KVStorageHttpServer` (wraps any KV storage;
|
||||
`handle(Request) → Response`): `POST /<primitive>` with Extended JSON bodies; `scan` streams
|
||||
NDJSON. Unreachable server (fetch failure, 502/503/504) → `StorageUnavailableError`. The server
|
||||
exposes the whole underlying store (primitives use absolute keys).
|
||||
- `KVStorageComposite`: see 4.1.
|
||||
- **Change stream (SSE).** `GET /events` on the server streams every change to the wrapped storage
|
||||
as `event: change` with id `<epoch>-<seq>` (epoch = server instance). The server keeps the last
|
||||
`historySize` (10 000) change batches.
|
||||
- A client reconnecting with `Last-Event-ID` gets the batches it missed, or `event: resync` if it
|
||||
can't be resumed (another epoch, or too old).
|
||||
- Every catch-up ends with `event: ready`.
|
||||
- Keep-alive comments are sent every 15 s.
|
||||
- `KVStorageHttp.watch()` subscribes (via the vendored `SSESession`, which reconnects with
|
||||
backoff), emits `change` events with `origin: 'remote'` and `resync`, and resolves once caught up.
|
||||
- Limitation: a server only sees writes made through its own storage instance, so several server
|
||||
processes over one database need a shared pub/sub (not built).
|
||||
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
|
||||
Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be
|
||||
encrypted freely.
|
||||
|
||||
### 4.1 `KVStorageComposite({ source, replicas, retry? })`
|
||||
|
||||
Keeps full local copies of a source (e.g. `KVStorageHttp`) so reads are local and the app works
|
||||
offline. Replicas are listed fastest first (e.g. `[inMemory, indexedDb]`). The last replica is the
|
||||
persistent one: it seeds the others on start and stores the composite's internal keys under `~q/`
|
||||
(hidden from reads): `~q/meta` (`{ lastEventId }`, present once a sync has completed) and the
|
||||
outbox `~q/o/<seq:12>`.
|
||||
|
||||
- **Start** (lazy, on first use):
|
||||
1. Load the outbox and copy the persistent replica into the faster ones.
|
||||
2. If a previous sync exists, serve reads locally straight away.
|
||||
3. Watch the source, resuming from `lastEventId`.
|
||||
4. Run a full sync only when there's no resume point, when the source sends `resync`, or when the
|
||||
source can't be watched.
|
||||
5. Before the first sync completes, reads go to the source.
|
||||
- **Remote changes** (and full syncs) are applied in order through one queue. A change that arrives
|
||||
during a full sync waits behind it, so the end state always matches the source. `lastEventId` is
|
||||
saved after each applied batch. Changes the replicas already hold are skipped.
|
||||
- **Full sync** merge-scans the source against the persistent replica (both key-ordered) and applies
|
||||
the difference to every replica in batches of 500.
|
||||
- **Writes** go to the source, then every replica.
|
||||
- On `StorageUnavailableError` (or while anything is queued, to keep order), the write is applied
|
||||
locally and appended to the outbox. The outbox is replayed in order with exponential backoff,
|
||||
and also whenever the source shows signs of life. Last writer wins on conflicts.
|
||||
- Remote changes to keys with queued writes are ignored until those writes are replayed.
|
||||
- `writeIfAbsent` always needs the source.
|
||||
- **Echo tracking.** Each write sent to a watched source expects one echo per key. A remote change
|
||||
to a key with an outstanding echo is consumed instead of applied. The stream is in server order,
|
||||
so that change is either the echo or older than it, and this avoids briefly showing a stale value.
|
||||
A full sync clears the counts.
|
||||
- Write through the composite, not directly through the same source instance (its `local`-origin
|
||||
changes are not mirrored).
|
||||
- `start()`, `synced()` (complete copy, all received changes applied), `queued` (outbox size),
|
||||
`close()`.
|
||||
|
||||
## 5. Event storage
|
||||
|
||||
`BaseEventStorage` primitives over stream paths: `appendTo`, `readFrom`, `versionOf`, `streamsOf`.
|
||||
@@ -153,6 +222,10 @@ Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)`
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
- Composite: replicate a subtree instead of the whole store; conflict policies beyond last writer
|
||||
wins for offline writes.
|
||||
- Multi-process servers need a shared change feed (pub/sub) for SSE.
|
||||
|
||||
- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The
|
||||
encryption wrapper could support per-subject keys.
|
||||
- Order-preserving numeric keys (`9` before `10`) as an opt-in key encoding.
|
||||
|
||||
Reference in New Issue
Block a user