# PotentialStorage — Design

Layered TypeScript storage: **KV → Events → Documents**. Each layer is an abstract base class
that implementations extend; each is usable on its own. No runtime dependencies.

This supersedes the original implementation brief (see git history before the `simplify` branch);
section 7 lists what changed and why.

## 1. Principles

1. **One abstract class per layer.** Implementations provide a handful of raw primitives; the
   whole user-facing API is written once in the base class. A client (e.g. HTTP) and a server-side
   store are the same kind of object, and wrappers (e.g. encryption) are just another subclass
   that delegates the primitives.
2. **One navigation primitive: `child(key)`.** Every instance is a node; `child` returns a view of
   the same storage rooted deeper (via `Object.create`, so it shares the instance's state). Sync, no I/O.
   Child nodes are **frozen**: an implementation that reassigns a field while running on a child
   would silently shadow the root's value, so that throws instead. Implementations keep mutable
   state in a `readonly state` object or in Maps.
3. **Self or child by argument.** `get()` reads this node, `get(key)` a child. A key that is passed
   but `undefined`/`null` throws (`InvalidKeyError`) instead of silently meaning "self". Methods use
   optional parameters or tuple-union rest parameters, not overloads.
4. **Indexes belong to the store.** Document stores receive the fields to index and use them however
   suits the backend (in-memory equality maps, native Mongo/SQLite/IndexedDB indexes, or nothing).
5. **Simple over clever.** Code style: classes, TS `private` (not `#`), no method overrides in
   subclasses, tests only under `test/`.
6. **No runtime dependencies.** Third-party code is vendored into `src/utils/` with a header naming
   its source: `sse-session/` (Harvmaster/sse-session), `event-emitter.ts` and `extended-json.ts`
   (xo-cash-utils, branch installable-sse). Node ≥ 22; CI runs Node 22, 24 and 26.

## 2. Layout

```
src/
  kv-storage/        base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts,
                     kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts
  event-storage/     base-event-storage.ts, event-storage-kv.ts
  document-storage/  base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
                     event-sourced-document-storage.ts
  utils/             keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types),
                     document-path/query/update, sse-session/ (vendored)
test/                mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
bench/
```

## 3. Keys (`utils/keys.ts`)

- A node's key is its escaped path segments joined by `/` (`users/123`). The separator is fixed.
- **Escaping:** every character at or below `/` (space, `!"#$%&'()*+,-.`, `/`, control characters)
  and `~` itself becomes `~XX` (uppercase hex). Escaped segments contain only characters above `/`,
  so all keys of one child sort together (`p/a`, `p/a/…`, then `p/a~2D1` for `a-1`). That keeps
  child listing a plain skip-scan.
- **Reserved segments** are `~` + a lowercase letter (`~e` = event commits). Escaping only produces
  `~` + uppercase hex, so user keys can't collide.
- Keys sort lexicographically by escaped form (`10` before `9`; `a-1` → `a~2D1` after `a0`).
- Empty keys, `NaN`/`Infinity` and non-string/number keys are rejected.

## 4. KV storage

`BaseKVStorage` primitives (absolute keys). Implementations provide the reads and the protected
`apply*` writes; the public `writeMany` / `removeMany` / `writeIfAbsent` live in the base class and
emit a `change` event after each write, so every store emits change events.

| Primitive                                              |                                                                                                      |
| ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| `readMany(keys)`                                       | values in order, `undefined` for missing                                                             |
| `scan({ gte, lt, limit, reverse, values })`            | key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once) |
| `applyWrite(entries)` → public `writeMany`             |                                                                                                      |
| `applyRemove(keys)` → public `removeMany`              |                                                                                                      |
| `applyWriteIfAbsent(entries)` → public `writeIfAbsent` | `true` = written; atomic per key (needed by events)                                                  |

**Values** are JSON plus `bigint` and `Uint8Array` (as in JSON, `-0` becomes `0` and `undefined` properties are dropped). Anything crossing the network is encoded as
Extended JSON (`<bigint: 123n>`, `<uint8array: abcd>`); IndexedDB stores them natively.

**Events** (`EventEmitter`, shared by all nodes of a storage): `change` →
`{ changes: { key, value? }[], origin: 'local' | 'remote', id? }` (no `value` = removed), and
`resync` → changes may have been missed, so reload.

Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `set(key, value)`,
`setMany`, `delete(key?)`, `deleteMany`, `deleteTree(key?)`, `keys({ limit, after, reverse })`,
`entries(...)`. The root has no own value. A node's own value and its children are independent keys.

- **`keys()`** is a skip-scan: one seek per child that has descendants; siblings without
  descendants share one iterator. It lists children that only have descendants, and skips
  reserved segments.
- **The only KV "index" is the ordered key space.** There is no `find`: with KV you know where to look,
  or you list a node's children.
- **Implementations:**
  - `KVStorageInMemory`: `SortedMap` (sorted chunks of ≤1024 keys); values deep-copied.
  - `KVStorageIndexedDB`: one object store with string keys. Scans read in batches, each in its own
    transaction, resuming after the last key (IndexedDB transactions close when the code awaits
    anything else). `writeIfAbsent` runs in one readwrite transaction.
  - `KVStorageHttp` (client) and `KVStorageHttpServer` (wraps any KV storage;
    `handle(Request) → Response`): `POST /<primitive>` with Extended JSON bodies; `scan` streams
    NDJSON. Unreachable server (fetch failure, 502/503/504) → `StorageUnavailableError`. The server
    exposes the whole underlying store (primitives use absolute keys).
  - `KVStorageComposite`: see 4.1.
- **Change stream (SSE).** `GET /events` on the server streams every change to the wrapped storage
  as `event: change` with id `<epoch>-<seq>` (epoch = server instance). The server keeps the last
  `historySize` (10 000) change batches.
  - A client reconnecting with `Last-Event-ID` gets the batches it missed, or `event: resync` if it
    can't be resumed (another epoch, or too old).
  - Every catch-up ends with `event: ready`.
  - Keep-alive comments are sent every 15 s.
  - `KVStorageHttp.watch()` subscribes (via the vendored `SSESession`, which reconnects with
    backoff), emits `change` events with `origin: 'remote'` and `resync`, and resolves once caught up.
  - Limitation: a server only sees writes made through its own storage instance, so several server
    processes over one database need a shared pub/sub (not built).
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
  Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be
  encrypted freely.

### 4.1 `KVStorageComposite({ source, replicas, retry? })`

Keeps full local copies of a source (e.g. `KVStorageHttp`) so reads are local and the app works
offline. Replicas are listed fastest first (e.g. `[inMemory, indexedDb]`). The last replica is the
persistent one: it seeds the others on start and stores the composite's internal keys under `~q/`
(hidden from reads): `~q/meta` (`{ lastEventId }`, present once a sync has completed) and the
outbox `~q/o/<seq:12>`.

- **Start** (lazy, on first use):
  1. Load the outbox and copy the persistent replica into the faster ones.
  2. If a previous sync exists, serve reads locally straight away.
  3. Watch the source, resuming from `lastEventId`.
  4. Run a full sync only when there's no resume point, when the source sends `resync`, or when the
     source can't be watched.
  5. Before the first sync completes, reads go to the source.
- **Remote changes** (and full syncs) are applied in order through one queue. A change that arrives
  during a full sync waits behind it, so the end state always matches the source. `lastEventId` is
  saved after each applied batch. Changes the replicas already hold are skipped.
- **Full sync** merge-scans the source against the persistent replica (both key-ordered) and applies
  the difference to every replica in batches of 500.
- **Writes** go to the source, then every replica.
  - On `StorageUnavailableError` (or while anything is queued, to keep order), the write is applied
    locally and appended to the outbox. The outbox is replayed in order with exponential backoff,
    and also whenever the source shows signs of life. Last writer wins on conflicts.
  - Remote changes to keys with queued writes are ignored until those writes are replayed.
  - `writeIfAbsent` always needs the source.
- **Echo tracking.** Each write sent to a watched source expects one echo per key. A remote change
  to a key with an outstanding echo is consumed instead of applied. The stream is in server order,
  so that change is either the echo or older than it, and this avoids briefly showing a stale value.
  A full sync clears the counts.
- Write through the composite, not directly through the same source instance (its `local`-origin
  changes are not mirrored).
- `start()`, `synced()` (complete copy, all received changes applied), `queued` (outbox size),
  `close()`.

## 5. Event storage

`BaseEventStorage` primitives over stream paths: `appendTo`, `readFrom`, `versionOf`, `streamsOf`.
Node API: `append(events, opts?)` / `append(key, events, opts?)`, `read(opts?)` / `read(key, opts?)`,
`version(key?)`, `keys()`. The root is not a stream. An empty stream is version 0; the first event is 1.
`expectedVersion: number | 'any'` (default `'any'`).

**`EventStorageKV`** (on any KV storage):

- One key per append, densely numbered: `<stream>/~e/<commit:12>` → `{ firstVersion, at, events }`.
  Keying by commit number (not version) means any writer with a stale view collides with the
  existing key.
- Append: head from a bounded cache (else one reverse scan, `limit: 1`) → check `expectedVersion`
  (re-reading the head before rejecting, in case the cache is stale) → `writeIfAbsent`. On a lost race:
  `ConcurrencyError` if a version was expected, otherwise retry (default 5).
- `read({ from })`: commit `n` starts at a version ≥ n, so try commit `from` first (exact for
  single-event commits), else binary-search; then forward-scan.
- Multi-writer safe on any store, because `writeIfAbsent` is a required primitive.

## 6. Document storage

`BaseDocumentStorage` primitives over document paths:

| Primitive                      |                                                                                                                                        |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| `readMany(paths)`              |                                                                                                                                        |
| `mutate(path, change)`         | maps current state → `DocChange` (or `null` = no-op), applies it, returns the new state; the implementation decides how it's made safe |
| `query(parent, DocumentQuery)` | children of `parent`, filtered, sorted and paged                                                                                       |

Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)` → ULID-like id,
`update(spec, opts?)` / `update(key, spec, opts?)` → new state or `undefined` (`{ upsert }`),
`delete(key?)` → existed, `find(filter, { sort, limit, skip })`, `findOne`, `keys()`.

- **Queries** (`utils/document-query.ts`): `{ field: value }` (dot paths; `null` also matches missing),
  `$eq $gt $gte $lt $lte $in`. Comparisons are type-bracketed. On arrays a condition matches the array
  or any one element. Sorting order: missing/null < booleans < numbers < strings < other.
- **Updates** (`utils/document-update.ts`): `$set $unset $inc $push $pull $addToSet` (`$each` for
  push/addToSet). They resolve to a `DocChange` with final values (`$inc` → `set: { n: 6 }`), so
  changes replay deterministically.
- **Indexes:** `{ indexes: { users: ['email', 'age'], 'users/*/posts': ['createdAt'] } }`, passed to
  the implementation.

**Implementations**

- `DocumentStorageInMemory`: maps per collection; equality index per declared field (used for
  `{ f: v }` / `{ f: { $eq: v } }`), otherwise scans. Writes are atomic in-process.
- `DocumentStorageKV`: each document is the KV value at its path; queries scan the collection's
  children; no indexes; concurrent writes to one document are last-writer-wins.
- `EventSourcedDocumentStorage(events, projections)`: each document's `DocChange`s are appended to
  its own stream. Current state lives in **any** `BaseDocumentStorage` used as the projection store,
  which serves all reads, queries and indexes.
  - Projections carry `_version` (reserved; stripped from results), used as `expectedVersion` on the
    next append.
  - If a write crashes between append and projection update, the next write conflicts, rebuilds the
    projection from the stream, and retries. `catchUp(key?)` rebuilds on demand.
  - Missing projections with stream history (deleted documents, crashed first writes) are rebuilt
    before writing.
  - Writes to one document are serialised in-process, so retries only happen across processes.
  - Extras: `history(key?)`, `getAt(version)` / `getAt(key, version)`. Children keep these methods
    in their types through interface declaration merging (types only).

## 7. Changes from the original brief

- Drivers + node classes → one abstract class per layer (client and server share an API).
- KV indexes (`findBy`, pattern trie, order-preserving value encoding, read-repair) removed.
  Document indexes are declared as fields and handled by each store; unique indexes and the query
  planner were removed.
- Queries: `$ne $nin $exists $and $or $not` and projections dropped.
- Separator fixed to `/`, and every character ≤ `/` is escaped so child keys stay contiguous. This
  removed the subtree-order workaround in child listing.
- Event-sourced documents are events + a projection store instead of a bespoke projection format.
- `writeIfAbsent` is a required primitive (no single-writer mutex fallback; no optional `batch` /
  `deleteRange`).
- `tree()` removed (use `entries()` for field-per-key reads).

## 8. Open questions

- Composite: replicate a subtree instead of the whole store; conflict policies beyond last writer
  wins for offline writes.
- Multi-process servers need a shared change feed (pub/sub) for SSE.

- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The
  encryption wrapper could support per-subject keys.
- Order-preserving numeric keys (`9` before `10`) as an opt-in key encoding.

## 9. Benchmark baselines

`npm run bench`, in-memory stores, Node 22, Linux x64 cloud container (shared CPU, ±10–20%). Mean per operation.

| Benchmark                                                            | Mean            |
| -------------------------------------------------------------------- | --------------- |
| KV `get` / `set` @ 10k keys                                          | 1.5 / 1.7 µs    |
| KV `get` / `set` @ 100k keys                                         | 2.0 / 2.3 µs    |
| KV `getMany` × 100                                                   | 25–35 µs        |
| `keys()` on 1k children × 100 descendants (skip-scan vs full scan)   | 2.8 ms vs 34 ms |
| Event `append` (warm cache)                                          | 7.8 µs          |
| Event `read` 100 from version 5 000 of 10k                           | 131 µs          |
| Event `read` full 10k stream                                         | 18 ms           |
| Documents in-memory @ 10k: `get` / `update`                          | 1.8 / 6.6 µs    |
| Documents in-memory: `find` indexed (~100 hits) / scan (~200 of 10k) | 0.20 / 11 ms    |
| Event-sourced @ 10k: `get` / `update`                                | 1.6 / 17 µs     |
| Event-sourced: `find` indexed / scan                                 | 0.38 / 12 ms    |