Claude
df7de27e20
A write made during a full sync could be scanned in the replica but not in the (read-ahead) source scan, so the diff removed it, and echo tracking then consumed the server's echo. Keys written during a sync are now left out of the diff. Deterministic regression test with a read-ahead source. Also exclude -0 from the Extended JSON property test (JSON has no -0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
250 lines
17 KiB
Markdown
250 lines
17 KiB
Markdown
# PotentialStorage — Design
|
||
|
||
Layered TypeScript storage: **KV → Events → Documents**. Each layer is an abstract base class
|
||
that implementations extend; each is usable on its own. No runtime dependencies.
|
||
|
||
This supersedes the original implementation brief (see git history before the `simplify` branch);
|
||
section 7 lists what changed and why.
|
||
|
||
## 1. Principles
|
||
|
||
1. **One abstract class per layer.** Implementations provide a handful of raw primitives; the
|
||
whole user-facing API is written once in the base class. A client (e.g. HTTP) and a server-side
|
||
store are the same kind of object, and wrappers (e.g. encryption) are just another subclass
|
||
that delegates the primitives.
|
||
2. **One navigation primitive: `child(key)`.** Every instance is a node; `child` returns a view of
|
||
the same storage rooted deeper (via `Object.create`, so it shares the instance's state). Sync, no I/O.
|
||
Child nodes are **frozen**: an implementation that reassigns a field while running on a child
|
||
would silently shadow the root's value, so that throws instead. Implementations keep mutable
|
||
state in a `readonly state` object or in Maps.
|
||
3. **Self or child by argument.** `get()` reads this node, `get(key)` a child. A key that is passed
|
||
but `undefined`/`null` throws (`InvalidKeyError`) instead of silently meaning "self". Methods use
|
||
optional parameters or tuple-union rest parameters, not overloads.
|
||
4. **Indexes belong to the store.** Document stores receive the fields to index and use them however
|
||
suits the backend (in-memory equality maps, native Mongo/SQLite/IndexedDB indexes, or nothing).
|
||
5. **Simple over clever.** Code style: classes, TS `private` (not `#`), no method overrides in
|
||
subclasses, tests only under `test/`.
|
||
6. **No runtime dependencies.** Third-party code is vendored into `src/utils/` with a header naming
|
||
its source: `sse-session/` (Harvmaster/sse-session), `event-emitter.ts` and `extended-json.ts`
|
||
(xo-cash-utils, branch installable-sse). Node ≥ 22; CI runs Node 22, 24 and 26.
|
||
|
||
## 2. Layout
|
||
|
||
```
|
||
src/
|
||
kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts,
|
||
kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts
|
||
event-storage/ base-event-storage.ts, event-storage-kv.ts
|
||
document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
|
||
event-sourced-document-storage.ts
|
||
utils/ keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types),
|
||
document-path/query/update, sse-session/ (vendored)
|
||
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
|
||
bench/
|
||
```
|
||
|
||
## 3. Keys (`utils/keys.ts`)
|
||
|
||
- A node's key is its escaped path segments joined by `/` (`users/123`). The separator is fixed.
|
||
- **Escaping:** every character at or below `/` (space, `!"#$%&'()*+,-.`, `/`, control characters)
|
||
and `~` itself becomes `~XX` (uppercase hex). Escaped segments contain only characters above `/`,
|
||
so all keys of one child sort together (`p/a`, `p/a/…`, then `p/a~2D1` for `a-1`). That keeps
|
||
child listing a plain skip-scan.
|
||
- **Reserved segments** are `~` + a lowercase letter (`~e` = event commits). Escaping only produces
|
||
`~` + uppercase hex, so user keys can't collide.
|
||
- Keys sort lexicographically by escaped form (`10` before `9`; `a-1` → `a~2D1` after `a0`).
|
||
- Empty keys, `NaN`/`Infinity` and non-string/number keys are rejected.
|
||
|
||
## 4. KV storage
|
||
|
||
`BaseKVStorage` primitives (absolute keys). Implementations provide the reads and the protected
|
||
`apply*` writes; the public `writeMany` / `removeMany` / `writeIfAbsent` live in the base class and
|
||
emit a `change` event after each write, so every store emits change events.
|
||
|
||
| Primitive | |
|
||
| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
|
||
| `readMany(keys)` | values in order, `undefined` for missing |
|
||
| `scan({ gte, lt, limit, reverse, values })` | key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once) |
|
||
| `applyWrite(entries)` → public `writeMany` | |
|
||
| `applyRemove(keys)` → public `removeMany` | |
|
||
| `applyWriteIfAbsent(entries)` → public `writeIfAbsent` | `true` = written; atomic per key (needed by events) |
|
||
|
||
**Values** are JSON plus `bigint` and `Uint8Array` (as in JSON, `-0` becomes `0` and `undefined` properties are dropped). Anything crossing the network is encoded as
|
||
Extended JSON (`<bigint: 123n>`, `<uint8array: abcd>`); IndexedDB stores them natively.
|
||
|
||
**Events** (`EventEmitter`, shared by all nodes of a storage): `change` →
|
||
`{ changes: { key, value? }[], origin: 'local' | 'remote', id? }` (no `value` = removed), and
|
||
`resync` → changes may have been missed, so reload.
|
||
|
||
Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `set(key, value)`,
|
||
`setMany`, `delete(key?)`, `deleteMany`, `deleteTree(key?)`, `keys({ limit, after, reverse })`,
|
||
`entries(...)`. The root has no own value. A node's own value and its children are independent keys.
|
||
|
||
- **`keys()`** is a skip-scan: one seek per child that has descendants; siblings without
|
||
descendants share one iterator. It lists children that only have descendants, and skips
|
||
reserved segments.
|
||
- **The only KV "index" is the ordered key space.** There is no `find`: with KV you know where to look,
|
||
or you list a node's children.
|
||
- **Implementations:**
|
||
- `KVStorageInMemory`: `SortedMap` (sorted chunks of ≤1024 keys); values deep-copied.
|
||
- `KVStorageIndexedDB`: one object store with string keys. Scans read in batches, each in its own
|
||
transaction, resuming after the last key (IndexedDB transactions close when the code awaits
|
||
anything else). `writeIfAbsent` runs in one readwrite transaction.
|
||
- `KVStorageHttp` (client) and `KVStorageHttpServer` (wraps any KV storage;
|
||
`handle(Request) → Response`): `POST /<primitive>` with Extended JSON bodies; `scan` streams
|
||
NDJSON. Unreachable server (fetch failure, 502/503/504) → `StorageUnavailableError`. The server
|
||
exposes the whole underlying store (primitives use absolute keys).
|
||
- `KVStorageComposite`: see 4.1.
|
||
- **Change stream (SSE).** `GET /events` on the server streams every change to the wrapped storage
|
||
as `event: change` with id `<epoch>-<seq>` (epoch = server instance). The server keeps the last
|
||
`historySize` (10 000) change batches.
|
||
- A client reconnecting with `Last-Event-ID` gets the batches it missed, or `event: resync` if it
|
||
can't be resumed (another epoch, or too old).
|
||
- Every catch-up ends with `event: ready`.
|
||
- Keep-alive comments are sent every 15 s.
|
||
- `KVStorageHttp.watch()` subscribes (via the vendored `SSESession`, which reconnects with
|
||
backoff), emits `change` events with `origin: 'remote'` and `resync`, and resolves once caught up.
|
||
- Limitation: a server only sees writes made through its own storage instance, so several server
|
||
processes over one database need a shared pub/sub (not built).
|
||
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
|
||
Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be
|
||
encrypted freely.
|
||
|
||
### 4.1 `KVStorageComposite({ source, replicas, retry? })`
|
||
|
||
Keeps full local copies of a source (e.g. `KVStorageHttp`) so reads are local and the app works
|
||
offline. Replicas are listed fastest first (e.g. `[inMemory, indexedDb]`). The last replica is the
|
||
persistent one: it seeds the others on start and stores the composite's internal keys under `~q/`
|
||
(hidden from reads): `~q/meta` (`{ lastEventId }`, present once a sync has completed) and the
|
||
outbox `~q/o/<seq:12>`.
|
||
|
||
- **Start** (lazy, on first use):
|
||
1. Load the outbox and copy the persistent replica into the faster ones.
|
||
2. If a previous sync exists, serve reads locally straight away.
|
||
3. Watch the source, resuming from `lastEventId`.
|
||
4. Run a full sync only when there's no resume point, when the source sends `resync`, or when the
|
||
source can't be watched.
|
||
5. Before the first sync completes, reads go to the source.
|
||
- **Remote changes** (and full syncs) are applied in order through one queue. A change that arrives
|
||
during a full sync waits behind it, so the end state always matches the source. `lastEventId` is
|
||
saved after each applied batch. Changes the replicas already hold are skipped.
|
||
- **Full sync** merge-scans the source against the persistent replica (both key-ordered) and applies
|
||
the difference to every replica in batches of 500.
|
||
- **Writes** go to the source, then every replica.
|
||
- On `StorageUnavailableError` (or while anything is queued, to keep order), the write is applied
|
||
locally and appended to the outbox. The outbox is replayed in order with exponential backoff,
|
||
and also whenever the source shows signs of life. Last writer wins on conflicts.
|
||
- Remote changes to keys with queued writes are ignored until those writes are replayed.
|
||
- `writeIfAbsent` always needs the source.
|
||
- **Echo tracking.** Each write sent to a watched source expects one echo per key. A remote change
|
||
to a key with an outstanding echo is consumed instead of applied. The stream is in server order,
|
||
so that change is either the echo or older than it, and this avoids briefly showing a stale value.
|
||
A full sync clears the counts.
|
||
- Write through the composite, not directly through the same source instance (its `local`-origin
|
||
changes are not mirrored).
|
||
- `start()`, `synced()` (complete copy, all received changes applied), `queued` (outbox size),
|
||
`close()`.
|
||
|
||
## 5. Event storage
|
||
|
||
`BaseEventStorage` primitives over stream paths: `appendTo`, `readFrom`, `versionOf`, `streamsOf`.
|
||
Node API: `append(events, opts?)` / `append(key, events, opts?)`, `read(opts?)` / `read(key, opts?)`,
|
||
`version(key?)`, `keys()`. The root is not a stream. An empty stream is version 0; the first event is 1.
|
||
`expectedVersion: number | 'any'` (default `'any'`).
|
||
|
||
**`EventStorageKV`** (on any KV storage):
|
||
|
||
- One key per append, densely numbered: `<stream>/~e/<commit:12>` → `{ firstVersion, at, events }`.
|
||
Keying by commit number (not version) means any writer with a stale view collides with the
|
||
existing key.
|
||
- Append: head from a bounded cache (else one reverse scan, `limit: 1`) → check `expectedVersion`
|
||
(re-reading the head before rejecting, in case the cache is stale) → `writeIfAbsent`. On a lost race:
|
||
`ConcurrencyError` if a version was expected, otherwise retry (default 5).
|
||
- `read({ from })`: commit `n` starts at a version ≥ n, so try commit `from` first (exact for
|
||
single-event commits), else binary-search; then forward-scan.
|
||
- Multi-writer safe on any store, because `writeIfAbsent` is a required primitive.
|
||
|
||
## 6. Document storage
|
||
|
||
`BaseDocumentStorage` primitives over document paths:
|
||
|
||
| Primitive | |
|
||
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| `readMany(paths)` | |
|
||
| `mutate(path, change)` | maps current state → `DocChange` (or `null` = no-op), applies it, returns the new state; the implementation decides how it's made safe |
|
||
| `query(parent, DocumentQuery)` | children of `parent`, filtered, sorted and paged |
|
||
|
||
Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)` → ULID-like id,
|
||
`update(spec, opts?)` / `update(key, spec, opts?)` → new state or `undefined` (`{ upsert }`),
|
||
`delete(key?)` → existed, `find(filter, { sort, limit, skip })`, `findOne`, `keys()`.
|
||
|
||
- **Queries** (`utils/document-query.ts`): `{ field: value }` (dot paths; `null` also matches missing),
|
||
`$eq $gt $gte $lt $lte $in`. Comparisons are type-bracketed. On arrays a condition matches the array
|
||
or any one element. Sorting order: missing/null < booleans < numbers < strings < other.
|
||
- **Updates** (`utils/document-update.ts`): `$set $unset $inc $push $pull $addToSet` (`$each` for
|
||
push/addToSet). They resolve to a `DocChange` with final values (`$inc` → `set: { n: 6 }`), so
|
||
changes replay deterministically.
|
||
- **Indexes:** `{ indexes: { users: ['email', 'age'], 'users/*/posts': ['createdAt'] } }`, passed to
|
||
the implementation.
|
||
|
||
**Implementations**
|
||
|
||
- `DocumentStorageInMemory`: maps per collection; equality index per declared field (used for
|
||
`{ f: v }` / `{ f: { $eq: v } }`), otherwise scans. Writes are atomic in-process.
|
||
- `DocumentStorageKV`: each document is the KV value at its path; queries scan the collection's
|
||
children; no indexes; concurrent writes to one document are last-writer-wins.
|
||
- `EventSourcedDocumentStorage(events, projections)`: each document's `DocChange`s are appended to
|
||
its own stream. Current state lives in **any** `BaseDocumentStorage` used as the projection store,
|
||
which serves all reads, queries and indexes.
|
||
- Projections carry `_version` (reserved; stripped from results), used as `expectedVersion` on the
|
||
next append.
|
||
- If a write crashes between append and projection update, the next write conflicts, rebuilds the
|
||
projection from the stream, and retries. `catchUp(key?)` rebuilds on demand.
|
||
- Missing projections with stream history (deleted documents, crashed first writes) are rebuilt
|
||
before writing.
|
||
- Writes to one document are serialised in-process, so retries only happen across processes.
|
||
- Extras: `history(key?)`, `getAt(version)` / `getAt(key, version)`. Children keep these methods
|
||
in their types through interface declaration merging (types only).
|
||
|
||
## 7. Changes from the original brief
|
||
|
||
- Drivers + node classes → one abstract class per layer (client and server share an API).
|
||
- KV indexes (`findBy`, pattern trie, order-preserving value encoding, read-repair) removed.
|
||
Document indexes are declared as fields and handled by each store; unique indexes and the query
|
||
planner were removed.
|
||
- Queries: `$ne $nin $exists $and $or $not` and projections dropped.
|
||
- Separator fixed to `/`, and every character ≤ `/` is escaped so child keys stay contiguous. This
|
||
removed the subtree-order workaround in child listing.
|
||
- Event-sourced documents are events + a projection store instead of a bespoke projection format.
|
||
- `writeIfAbsent` is a required primitive (no single-writer mutex fallback; no optional `batch` /
|
||
`deleteRange`).
|
||
- `tree()` removed (use `entries()` for field-per-key reads).
|
||
|
||
## 8. Open questions
|
||
|
||
- Composite: replicate a subtree instead of the whole store; conflict policies beyond last writer
|
||
wins for offline writes.
|
||
- Multi-process servers need a shared change feed (pub/sub) for SSE.
|
||
|
||
- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The
|
||
encryption wrapper could support per-subject keys.
|
||
- Order-preserving numeric keys (`9` before `10`) as an opt-in key encoding.
|
||
|
||
## 9. Benchmark baselines
|
||
|
||
`npm run bench`, in-memory stores, Node 22, Linux x64 cloud container (shared CPU, ±10–20%). Mean per operation.
|
||
|
||
| Benchmark | Mean |
|
||
| -------------------------------------------------------------------- | --------------- |
|
||
| KV `get` / `set` @ 10k keys | 1.5 / 1.7 µs |
|
||
| KV `get` / `set` @ 100k keys | 2.0 / 2.3 µs |
|
||
| KV `getMany` × 100 | 25–35 µs |
|
||
| `keys()` on 1k children × 100 descendants (skip-scan vs full scan) | 2.8 ms vs 34 ms |
|
||
| Event `append` (warm cache) | 7.8 µs |
|
||
| Event `read` 100 from version 5 000 of 10k | 131 µs |
|
||
| Event `read` full 10k stream | 18 ms |
|
||
| Documents in-memory @ 10k: `get` / `update` | 1.8 / 6.6 µs |
|
||
| Documents in-memory: `find` indexed (~100 hits) / scan (~200 of 10k) | 0.20 / 11 ms |
|
||
| Event-sourced @ 10k: `get` / `update` | 1.6 / 17 µs |
|
||
| Event-sourced: `find` indexed / scan | 0.38 / 12 ms |
|