PotentialStorage/DESIGN.md

PotentialStorage — Design

Layered TypeScript storage: KV → Events → Documents. Each layer is an abstract base class that implementations extend; each is usable on its own. No runtime dependencies.

This supersedes the original implementation brief (see git history before the simplify branch); section 7 lists what changed and why.

1. Principles

1. One abstract class per layer. Implementations provide a handful of raw primitives; the whole user-facing API is written once in the base class. A client (e.g. HTTP) and a server-side store are the same kind of object, and wrappers (e.g. encryption) are just another subclass that delegates the primitives.
2. One navigation primitive: child(key). Every instance is a node; child returns a view of the same storage rooted deeper (via Object.create, so it shares the instance's state). Sync, no I/O. Child nodes are frozen: an implementation that reassigns a field while running on a child would silently shadow the root's value, so that throws instead. Implementations keep mutable state in a readonly state object or in Maps.
3. Self or child by argument. get() reads this node, get(key) a child. A key that is passed but undefined/null throws (InvalidKeyError) instead of silently meaning "self". Methods use optional parameters or tuple-union rest parameters, not overloads.
4. Indexes belong to the store. Document stores receive the fields to index and use them however suits the backend (in-memory equality maps, native Mongo/SQLite/IndexedDB indexes, or nothing).
5. Simple over clever. Code style: classes, TS private (not #), no method overrides in subclasses, tests only under test/.
6. No runtime dependencies. Third-party code is vendored into src/utils/ with a header naming its source: sse-session/ (Harvmaster/sse-session), event-emitter.ts and extended-json.ts (xo-cash-utils, branch installable-sse). Node ≥ 22; CI runs Node 22, 24 and 26.

2. Layout

src/
  kv-storage/        base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts,
                     kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts
  event-storage/     base-event-storage.ts, event-storage-kv.ts
  document-storage/  base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
                     event-sourced-document-storage.ts
  utils/             keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types),
                     document-path/query/update, sse-session/ (vendored)
test/                mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
bench/

3. Keys (utils/keys.ts)

• A node's key is its escaped path segments joined by / (users/123). The separator is fixed.
• Escaping: every character at or below / (space, !"#$%&'()*+,-., /, control characters) and ~ itself becomes ~XX (uppercase hex). Escaped segments contain only characters above /, so all keys of one child sort together (p/a, p/a/…, then p/a~2D1 for a-1). That keeps child listing a plain skip-scan.
• Reserved segments are ~ + a lowercase letter (~e = event commits). Escaping only produces ~ + uppercase hex, so user keys can't collide.
• Keys sort lexicographically by escaped form (10 before 9; a-1 → a~2D1 after a0).
• Empty keys, NaN/Infinity and non-string/number keys are rejected.

4. KV storage

BaseKVStorage primitives (absolute keys). Implementations provide the reads and the protected apply* writes; the public writeMany / removeMany / writeIfAbsent live in the base class and emit a change event after each write, so every store emits change events.

Primitive
readMany(keys)	values in order, undefined for missing
scan({ gte, lt, limit, reverse, values })	key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once)
applyWrite(entries) → public writeMany
applyRemove(keys) → public removeMany
applyWriteIfAbsent(entries) → public writeIfAbsent	true = written; atomic per key (needed by events)

Values are JSON plus bigint and Uint8Array (as in JSON, -0 becomes 0 and undefined properties are dropped). Anything crossing the network is encoded as Extended JSON (<bigint: 123n>, <uint8array: abcd>); IndexedDB stores them natively.

Events (EventEmitter, shared by all nodes of a storage): change → { changes: { key, value? }[], origin: 'local' | 'remote', id? } (no value = removed), and resync → changes may have been missed, so reload.

Node API (defined once): child, as, get(key?), getMany, set(value) / set(key, value), setMany, delete(key?), deleteMany, deleteTree(key?), keys({ limit, after, reverse }), entries(...). The root has no own value. A node's own value and its children are independent keys.

• keys() is a skip-scan: one seek per child that has descendants; siblings without descendants share one iterator. It lists children that only have descendants, and skips reserved segments.
• The only KV "index" is the ordered key space. There is no find: with KV you know where to look, or you list a node's children.
• Implementations:
  • KVStorageInMemory: SortedMap (sorted chunks of ≤1024 keys); values deep-copied.
  • KVStorageIndexedDB: one object store with string keys. Scans read in batches, each in its own transaction, resuming after the last key (IndexedDB transactions close when the code awaits anything else). writeIfAbsent runs in one readwrite transaction.
  • KVStorageHttp (client) and KVStorageHttpServer (wraps any KV storage; handle(Request) → Response): POST /<primitive> with Extended JSON bodies; scan streams NDJSON. Unreachable server (fetch failure, 502/503/504) → StorageUnavailableError. The server exposes the whole underlying store (primitives use absolute keys).
  • KVStorageComposite: see 4.1.
• Change stream (SSE). GET /events on the server streams every change to the wrapped storage as event: change with id <epoch>-<seq> (epoch = server instance). The server keeps the last historySize (10 000) change batches.
  • A client reconnecting with Last-Event-ID gets the batches it missed, or event: resync if it can't be resumed (another epoch, or too old).
  • Every catch-up ends with event: ready.
  • Keep-alive comments are sent every 15 s.
  • KVStorageHttp.watch() subscribes (via the vendored SSESession, which reconnects with backoff), emits change events with origin: 'remote' and resync, and resolves once caught up.
  • Limitation: a server only sees writes made through its own storage instance, so several server processes over one database need a shared pub/sub (not built).
• Wrappers (e.g. a future encryption layer) extend BaseKVStorage and delegate the primitives. Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be encrypted freely.

4.1 KVStorageComposite({ source, replicas, retry? })

Keeps full local copies of a source (e.g. KVStorageHttp) so reads are local and the app works offline. Replicas are listed fastest first (e.g. [inMemory, indexedDb]). The last replica is the persistent one: it seeds the others on start and stores the composite's internal keys under ~q/ (hidden from reads): ~q/meta ({ lastEventId }, present once a sync has completed) and the outbox ~q/o/<seq:12>.

• Start (lazy, on first use):
  1. Load the outbox and copy the persistent replica into the faster ones.
  2. If a previous sync exists, serve reads locally straight away.
  3. Watch the source, resuming from lastEventId.
  4. Run a full sync only when there's no resume point, when the source sends resync, or when the source can't be watched.
  5. Before the first sync completes, reads go to the source.
• Remote changes (and full syncs) are applied in order through one queue. A change that arrives during a full sync waits behind it, so the end state always matches the source. lastEventId is saved after each applied batch. Changes the replicas already hold are skipped.
• Full sync merge-scans the source against the persistent replica (both key-ordered) and applies the difference to every replica in batches of 500.
• Writes go to the source, then every replica.
  • On StorageUnavailableError (or while anything is queued, to keep order), the write is applied locally and appended to the outbox. The outbox is replayed in order with exponential backoff, and also whenever the source shows signs of life. Last writer wins on conflicts.
  • Remote changes to keys with queued writes are ignored until those writes are replayed.
  • writeIfAbsent always needs the source.
• Echo tracking. Each write sent to a watched source expects one echo per key. A remote change to a key with an outstanding echo is consumed instead of applied. The stream is in server order, so that change is either the echo or older than it, and this avoids briefly showing a stale value. A full sync clears the counts.
• Write through the composite, not directly through the same source instance (its local-origin changes are not mirrored).
• start(), synced() (complete copy, all received changes applied), queued (outbox size), close().

5. Event storage

BaseEventStorage primitives over stream paths: appendTo, readFrom, versionOf, streamsOf. Node API: append(events, opts?) / append(key, events, opts?), read(opts?) / read(key, opts?), version(key?), keys(). The root is not a stream. An empty stream is version 0; the first event is 1. expectedVersion: number | 'any' (default 'any').

EventStorageKV (on any KV storage):

• One key per append, densely numbered: <stream>/~e/<commit:12> → { firstVersion, at, events }. Keying by commit number (not version) means any writer with a stale view collides with the existing key.
• Append: head from a bounded cache (else one reverse scan, limit: 1) → check expectedVersion (re-reading the head before rejecting, in case the cache is stale) → writeIfAbsent. On a lost race: ConcurrencyError if a version was expected, otherwise retry (default 5).
• read({ from }): commit n starts at a version ≥ n, so try commit from first (exact for single-event commits), else binary-search; then forward-scan.
• Multi-writer safe on any store, because writeIfAbsent is a required primitive.

6. Document storage

BaseDocumentStorage primitives over document paths:

Primitive
readMany(paths)
mutate(path, change)	maps current state → DocChange (or null = no-op), applies it, returns the new state; the implementation decides how it's made safe
query(parent, DocumentQuery)	children of parent, filtered, sorted and paged

Node API: get(key?), getMany, set(doc) / set(key, doc), insert(doc) → ULID-like id, update(spec, opts?) / update(key, spec, opts?) → new state or undefined ({ upsert }), delete(key?) → existed, find(filter, { sort, limit, skip }), findOne, keys().

• Queries (utils/document-query.ts): { field: value } (dot paths; null also matches missing), $eq $gt $gte $lt $lte $in. Comparisons are type-bracketed. On arrays a condition matches the array or any one element. Sorting order: missing/null < booleans < numbers < strings < other.
• Updates (utils/document-update.ts): $set $unset $inc $push $pull $addToSet ($each for push/addToSet). They resolve to a DocChange with final values ($inc → set: { n: 6 }), so changes replay deterministically.
• Indexes: { indexes: { users: ['email', 'age'], 'users/*/posts': ['createdAt'] } }, passed to the implementation.

Implementations

• DocumentStorageInMemory: maps per collection; equality index per declared field (used for { f: v } / { f: { $eq: v } }), otherwise scans. Writes are atomic in-process.
• DocumentStorageKV: each document is the KV value at its path; queries scan the collection's children; no indexes; concurrent writes to one document are last-writer-wins.
• EventSourcedDocumentStorage(events, projections): each document's DocChanges are appended to its own stream. Current state lives in any BaseDocumentStorage used as the projection store, which serves all reads, queries and indexes.
  • Projections carry _version (reserved; stripped from results), used as expectedVersion on the next append.
  • If a write crashes between append and projection update, the next write conflicts, rebuilds the projection from the stream, and retries. catchUp(key?) rebuilds on demand.
  • Missing projections with stream history (deleted documents, crashed first writes) are rebuilt before writing.
  • Writes to one document are serialised in-process, so retries only happen across processes.
  • Extras: history(key?), getAt(version) / getAt(key, version). Children keep these methods in their types through interface declaration merging (types only).

7. Changes from the original brief

• Drivers + node classes → one abstract class per layer (client and server share an API).
• KV indexes (findBy, pattern trie, order-preserving value encoding, read-repair) removed. Document indexes are declared as fields and handled by each store; unique indexes and the query planner were removed.
• Queries: $ne $nin $exists $and $or $not and projections dropped.
• Separator fixed to /, and every character ≤ / is escaped so child keys stay contiguous. This removed the subtree-order workaround in child listing.
• Event-sourced documents are events + a projection store instead of a bespoke projection format.
• writeIfAbsent is a required primitive (no single-writer mutex fallback; no optional batch / deleteRange).
• tree() removed (use entries() for field-per-key reads).

8. Open questions

• Composite: replicate a subtree instead of the whole store; conflict policies beyond last writer wins for offline writes.

• Multi-process servers need a shared change feed (pub/sub) for SSE.

• Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The encryption wrapper could support per-subject keys.

• Order-preserving numeric keys (9 before 10) as an opt-in key encoding.

9. Benchmark baselines

npm run bench, in-memory stores, Node 22, Linux x64 cloud container (shared CPU, ±10–20%). Mean per operation.

Benchmark	Mean
KV get / set @ 10k keys	1.5 / 1.7 µs
KV get / set @ 100k keys	2.0 / 2.3 µs
KV getMany × 100	25–35 µs
keys() on 1k children × 100 descendants (skip-scan vs full scan)	2.8 ms vs 34 ms
Event append (warm cache)	7.8 µs
Event read 100 from version 5 000 of 10k	131 µs
Event read full 10k stream	18 ms
Documents in-memory @ 10k: get / update	1.8 / 6.6 µs
Documents in-memory: find indexed (~100 hits) / scan (~200 of 10k)	0.20 / 11 ms
Event-sourced @ 10k: get / update	1.6 / 17 µs
Event-sourced: find indexed / scan	0.38 / 12 ms