diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 4789154..b7c2b70 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -2,9 +2,12 @@ name: CI on: push: - branches: [main] + branches: ['**'] # not tags pull_request: +env: + NPM_REGISTRY: https://verdaccio.harvmaster.com/ + jobs: check: name: Node ${{ matrix.node }} @@ -12,7 +15,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22] + node: [22, 24, 26] steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 @@ -28,3 +31,41 @@ jobs: run: npm run test:only - name: Build run: npm run build + + browser: + name: Chromium + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Install Chromium + run: npx playwright install --with-deps chromium + - name: Browser tests + run: npm run test:browser + + # Pushes only (not pull requests): main → `latest`, next → `next`, other branches → `dev-`. + # See scripts/publish.mjs. Needs the NPM_REGISTRY_TOKEN secret. + staging: + name: Publish to staging registry + needs: [check, browser] + if: github.event_name == 'push' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Configure registry + run: | + host="${NPM_REGISTRY#https://}" + printf 'registry=%s\n//%s:_authToken=${NPM_REGISTRY_TOKEN}\n' "$NPM_REGISTRY" "$host" > .npmrc + - name: Publish + run: node scripts/publish.mjs "${{ github.ref_name }}" + env: + NPM_REGISTRY_TOKEN: ${{ secrets.NPM_REGISTRY_TOKEN }} diff --git a/CLAUDE.md b/CLAUDE.md index 8553941..4ec6aca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,24 +1,33 @@ # PotentialStorage -Layered TypeScript storage library: KV → Events → Documents. The full design, milestones and -implementation decisions are in [DESIGN.md](./DESIGN.md) — read it before changing anything. -Record new decisions in DESIGN.md section 12. +Layered TypeScript storage library: KV → Events → Documents. The design is in +[DESIGN.md](./DESIGN.md); read it before changing anything and keep it up to date. ## Commands -- `npm test` — `tsc --noEmit` then all Vitest tests (including `*.test-d.ts` type tests) -- `npm run test:only -- ` — Vitest only, e.g. a single file -- `npm run typecheck` — `tsc --noEmit` over src, test and bench -- `npm run bench` — `vitest bench` -- `npm run build` — emit `dist/` -- `npm run format` / `npm run format:check` — Prettier +- `npm test`: `tsc --noEmit`, then all Vitest tests (including `test/types.test-d.ts`) +- `npm run test:only -- `: Vitest only +- `npm run test:browser`: `test/browser/` in headless Chromium (Vitest browser mode + Playwright) + against a real server on another origin (`test/browser/global-setup.ts`) +- `npm run typecheck`, `npm run build`, `npm run bench` +- `npm run format` / `npm run format:check`: Prettier -CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs format check, typecheck, tests and build on Node 20 and 22. +CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs the format check, typecheck, tests and build on +Node 22, 24 and 26, and the Chromium tests. On pushes it then publishes to the staging registry +(`scripts/publish.mjs`: `main` → `latest`, `next` → `next`, others → `dev-`). -## Rules +## Conventions -- No runtime dependencies in `src/`. Dev dependencies only. -- Drivers stay flat and string-keyed; all path/index/escaping logic lives in the node classes. -- Self vs child dispatch is by `arguments.length`; a passed-but-`undefined`/`null` key throws. -- Multi-result reads are `AsyncIterable`; don't build large arrays internally. -- Keep tests green and commit per milestone. Ask the owner before pushing. +- One abstract base class per layer (`src//base-*.ts`); implementations only provide the raw + primitives, and the user-facing API lives in the base class. +- Classes over loose functions; TS `private`, not `#`; no overloads (use `param?` or tuple-union + rest params); don't override base methods in subclasses. +- **Never reassign a field after construction.** `child()` nodes are frozen `Object.create(root)` + views, so `this.x = ...` on a child throws. Keep mutable state in a `private readonly state` + object or in Maps (see `KVStorageComposite`, `KVStorageHttp`). +- No runtime dependencies; third-party code is vendored (`src/utils/sse-session/`, `event-emitter.ts`, + `extended-json.ts`, with a header naming the source). Tests live only in `test/`, mirroring `src/`. +- `README.md` examples are mirrored in `test/readme.test.ts`; change both together. +- New KV/document implementations must pass `test/kv-storage/kv-storage.conformance.ts` / + `test/document-storage/document-storage.suite.ts`. +- Ask the owner before pushing. diff --git a/DESIGN.md b/DESIGN.md index 93b0e3a..e50d9d1 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1,655 +1,321 @@ # PotentialStorage — Design -This document is the design brief agreed with the project owner (sections 0–11), followed by -implementation decisions made where the brief was silent (section 12) and benchmark baselines -(section 13). +Layered TypeScript storage: **KV → Events → Documents**. Each layer is an abstract base class +that implementations extend; each is usable on its own. No runtime dependencies. -## 0. Setup and workflow - -- Repository: `git@git.harvmaster.com:Harvmaster/PotentialStorage.git` (SSH; self-hosted Git, not GitHub). Clone it. If you can't reach it, stop and tell the owner rather than starting a fresh repo elsewhere. -- Inspect what's already in the repo first. If it has tooling (package.json, tsconfig, test runner), keep it. If it's empty, set up: - - TypeScript (`strict: true`, `noUncheckedIndexedAccess: true`), ESM, Node ≥ 20 - - Vitest for tests and `vitest bench` for benchmarks; `expectTypeOf` for type tests - - Package name `potential-storage` unless one already exists - - **No runtime dependencies in the core.** Dev dependencies are fine (e.g. `fast-check` for property tests). -- Copy this brief into the repo as `DESIGN.md` and create a short `CLAUDE.md` pointing at it plus the build/test commands. -- Work milestone by milestone (section 9). Commit after each milestone with a clear message, and keep tests green. Ask the owner before pushing. +This supersedes the original implementation brief (see git history before the `simplify` branch); +section 7 lists what changed and why. ## 1. Principles -1. **Three layers**, each usable on its own: **KV** → **Event** (append-only, built on KV) → **Document** (Mongo-like, built on Events or native). -2. **Drivers vs stores.** Backends implement a tiny, flat, string-keyed _driver_. All path, child, index, escaping and typing logic lives once in the user-facing _node_ classes, so every backend gets identical behaviour. -3. **One navigation primitive: `child(key)`.** There are no collection/document classes. Every node is a store over its direct children, and `child(key)` re-roots to that key. Navigation is synchronous and does no I/O. -4. **No methods on data.** Reads return plain values. -5. **Performance is a requirement.** Batch operations are the primitive; scans are ordered and streamed; nothing replays or scans more than it must (section 8). +1. **One abstract class per layer.** Implementations provide a handful of raw primitives; the + whole user-facing API is written once in the base class. A client (e.g. HTTP) and a server-side + store are the same kind of object, and wrappers (e.g. encryption) are just another subclass + that delegates the primitives. +2. **One navigation primitive: `child(key)`.** Every instance is a node; `child` returns a view of + the same storage rooted deeper (via `Object.create`, so it shares the instance's state). Sync, no I/O. + Child nodes are **frozen**: an implementation that reassigns a field while running on a child + would silently shadow the root's value, so that throws instead. Implementations keep mutable + state in a `readonly state` object or in Maps. +3. **Self or child by argument.** `get()` reads this node, `get(key)` a child. A key that is passed + but `undefined`/`null` throws (`InvalidKeyError`) instead of silently meaning "self". Methods use + optional parameters or tuple-union rest parameters, not overloads. +4. **Indexes belong to the store.** Document stores receive the fields to index and use them however + suits the backend (in-memory equality maps, native Mongo/SQLite/IndexedDB indexes, or nothing). +5. **Simple over clever.** Code style: classes, TS `private` (not `#`), no method overrides in + subclasses, tests only under `test/`. +6. **No runtime dependencies.** Third-party code is vendored into `src/utils/` with a header naming + its source: `sse-session/` (Harvmaster/sse-session), `event-emitter.ts` and `extended-json.ts` + (xo-cash-utils, branch installable-sse). Node ≥ 22; CI runs Node 22, 24 and 26. -## 2. Paths and key encoding (`src/core/path.ts`) - -- A node's path is `readonly string[]` of **unescaped** segments. Number keys are converted with `String()`. -- The separator is a single character, configurable, default `/`. -- **Escaping** (per segment): percent-encode `%`, the separator, and a **leading** `~`. Escaped segments never contain a raw separator. Write `escapeSegment` / `unescapeSegment` with round-trip property tests. -- **Reserved segments** start with `~` (user segments can't, because a leading `~` is escaped): - - `~i` index entries - - `~u` unique-index claims - - `~e` event commits - - `~p` document projections root -- Each node precomputes in its constructor: - - `selfKey` = escaped segments joined by the separator (e.g. `users/123`) - - `prefix` = `selfKey + sep` (`''` for the root) - - No string work per call beyond `prefix + escapeSegment(key)`. -- Keys are ordered **lexicographically** (so `10` sorts before `9`). Document this. An order-preserving numeric key encoding is a possible later option, not part of v1. - -Example layout: +## 2. Layout ``` -users/123 → KV value of users/123 -users/123/posts/p1 → KV value of a nested child -users/~i/byEmail/sada@x.io/123 → index entry (value: empty / true) -orders/o1/~e/000000000007 → event commit #7 of stream orders/o1 -~p/users/123 → document projection { state, version, commit } +src/ + kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts, + kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts, + kv-storage-encrypted.ts + event-storage/ base-event-storage.ts, event-storage-kv.ts + document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts, + event-sourced-document-storage.ts + utils/ keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types), + document-path/query/update, sse-session/ (vendored) +test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts; + browser/ runs in headless Chromium against a real cross-origin server; + readme.test.ts runs the README's examples +bench/ ``` -## 3. Layer 1: KV - -### 3.1 Driver interface (`src/kv/driver.ts`) - -```ts -export interface ScanOptions { - gte?: string - lt?: string - limit?: number - reverse?: boolean - values?: boolean // when false/omitted, yield value as undefined -} - -export interface KVDriver { - getMany(keys: string[]): Promise<(unknown | undefined)[]> // same order as keys - setMany(entries: [key: string, value: unknown][]): Promise - deleteMany(keys: string[]): Promise - scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]> // MUST be key-ordered - - // Optional capabilities. Stores feature-detect them and fall back where possible. - setIfAbsent?(entries: [string, unknown][]): Promise // true = written - batch?(ops: BatchOp[]): Promise // atomic multi-op write - deleteRange?(gte: string, lt: string): Promise -} - -export type BatchOp = { type: 'set'; key: string; value: unknown } | { type: 'delete'; key: string } -``` - -- Values are passed through as JSON-serialisable values. Drivers that need bytes (HTTP, Redis, files) serialise internally via a pluggable `Codec` (default JSON). -- Provide an abstract `BaseKVDriver` with default implementations of `deleteRange` (scan + deleteMany) and helpers, so a minimal driver implements only the four required methods. -- **`InMemoryKVDriver`**: a sorted structure (sorted array with binary search is fine for v1; keep it behind an interface so it can become a B-tree). It implements every optional capability: `setIfAbsent`, atomic `batch` and `deleteRange`. -- Write a reusable **driver conformance suite** `runKVDriverConformance(name, makeDriver)` covering ordering, `gte`/`lt` bounds, reverse, limit, `values` on/off, `getMany` order with missing keys, and each optional capability when present. - -### 3.2 `KVStorage` / `KVNode` (`src/kv/node.ts`) - -```ts -const db = new KVStorage(driver, { separator: '/', indexes: { ... } }) // db is the root KVNode -const users = db.child('users') -const user = users.child('123') -``` - -API of every `KVNode` (types in section 7): - -| Method | Meaning | -| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -| `child(key)` | Sync. Returns a node at `path + key`. | -| `as()` | Type assertion for untyped use; returns the same node retyped. | -| `get()` / `get(key)` | Own value at `selfKey` / value at child key. | -| `getMany(keys)` | Batch child read, same order, `undefined` for missing. | -| `set(value)` / `set(key, value)` | Write own value / child value. | -| `setMany(entries)` | Batch child write. | -| `delete()` / `delete(key)` | Delete own value / child value only (not descendants). | -| `deleteMany(keys)` | Batch child delete. | -| `deleteTree()` / `deleteTree(key)` | Delete own value **and all descendants** (uses `deleteRange` if available). | -| `keys(opts?)` | `AsyncIterable` of **direct** child keys (unescaped). Options: `limit`, `after`, `reverse`. | -| `entries(opts?)` | `AsyncIterable<[key, value]>` of direct children that have an own value. | -| `tree({ depth })` | Assemble an object from descendant keys up to `depth` (for field-per-key style). Bounded and explicit; never the default. | -| `findBy(index, value \| { gte?, lt? }, opts?)` | Index lookup over direct children (section 3.3). | - -**Arity dispatch (important):** - -- Self vs child is decided by `arguments.length`, never by checking for `undefined`. -- If a key argument is passed but is `undefined` or `null`, throw a `TypeError` (this prevents `users.get(maybeId)` silently reading self). -- The root node (`path = []`) has no own value: `get()`, `set(v)` and `delete()` on root throw. -- Implement one private `#resolve(args)` that every method uses. - -**Direct-children listing (`keys`)** uses a skip-scan: - -1. Scan `gte: prefix`, `lt: prefix + succ(sep)`. -2. For each key, take the first segment after `prefix`. Skip reserved (`~…`) segments. Yield the segment (unescaped) if it differs from the previous one. -3. When the next key is a descendant of the segment just yielded, re-seek the driver to `gte: prefix + escSeg + succ(sep)` (where `succ(c)` is the next char code) instead of iterating through the subtree. Keep using the same iterator while keys are siblings. - -Listing must be O(direct children × seek cost), not O(subtree). A child that has no own value but has descendants **is** listed. - -### 3.3 Indexes (`src/kv/indexes.ts`) - -Indexes are declared on **path patterns** in the `KVStorage` constructor. A pattern names a parent node, and its index applies to that node's direct children's values: - -```ts -new KVStorage(driver, { - indexes: { - users: { byEmail: (u: User) => u.email }, - 'users/*/posts': { byDate: (p: Post) => p.createdAt, byTag: (p: Post) => p.tags }, - }, -}) -``` - -- An index function returns `IndexValue | IndexValue[] | undefined`, where `IndexValue = string | number | boolean | null`. An array means multi-entry; `undefined` means not indexed. -- Compile patterns into a **segment trie** (`*` matches one segment). A node resolves its index set **once, at construction**, in O(depth). -- Entry key: `~i///` with a `true` value. -- **`encodeIndexValue` must be order-preserving and separator-safe:** - - Type tag first, so values order by type: `null` < `false` < `true` < numbers < strings. - - Numbers use an order-preserving float64 encoding (flip the sign bit for positives and all bits for negatives, then fixed-width hex). - - Strings are escaped like segments. - - This enables `findBy(index, { gte, lt })` range scans. -- **Writes to an indexed node:** - 1. Read the old value(s) (`getMany`) and compute the diff of index entries. - 2. If the driver has `batch`, write the value and the index adds/removes atomically. - 3. Otherwise, write index **adds first**, then the value, then index **removes**. Treat index entries as hints: `findBy` re-reads each value, re-checks the index function, and lazily deletes stale entries (read-repair). -- Non-indexed nodes must **not** pay the read-before-write cost. - -## 4. Layer 2: Events (`src/events/`) - -An append-only log per stream. A stream is identified by a node path. Same `child()` navigation as KV. - -```ts -const events = new EventStorage(kv) // takes a KVStorage (or a KVNode as root) -const orders = events.child('orders') - -await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 }) -await orders.child('o1').append([{ type: 'paid' }]) // self form -for await (const e of orders.read('o1', { from: 1 })) { - /* { version, commit, at, data } */ -} -await orders.version('o1') -for await (const id of orders.keys()) { -} // child streams -orders.child('o1').child('shipments') -``` - -- `EventStorage` is the abstract interface. `KVEventStorage` is the KV-backed implementation. Leave room for native backends (EventStoreDB, Kafka) implementing the same interface. -- **Arity:** `append(events, opts?)` vs `append(key, events, opts?)`. Dispatch on `Array.isArray(args[0])`, plus the undefined-key rule from 3.2. `read`, `version` and `keys` follow the 3.2 rules. -- **Versions:** an empty stream has version 0, and the first event has version 1. `expectedVersion: number | 'any'` (default `'any'`). -- **Storage: one key per commit, densely numbered.** - - Key: `/~e/` - - Value: `{ firstVersion, at, events: E[] }` - - The commit sequence (not the event version) is the key so that **any** stale writer collides with the existing key. Keying by version would let a stale writer create overlapping version ranges. -- **Append:** - 1. Get the head `{ commitSeq, version }` (from the cache, or a reverse scan with `limit: 1`). - 2. If `expectedVersion` is a number and doesn't match the head, throw `ConcurrencyError`. - 3. Call `setIfAbsent(nextCommitKey)`. - 4. On `false` (collision): refresh the head. If `expectedVersion` was given, throw `ConcurrencyError`; otherwise retry up to 5 times. - 5. Without `setIfAbsent`, fall back to a per-stream in-process mutex. Document that this mode is single-writer only. -- **Head cache:** a bounded LRU keyed by stream path. Update it on successful append; invalidate it on collision. -- **`read({ from, to, reverse, limit })`** streams events with metadata. To start at version `from`, locate the containing commit by binary search over commit numbers using `getMany` probes (O(log commits) round trips), then forward-scan. -- There is no update or delete. Keys inside `~e` are never rewritten. - -## 5. Layer 3: Documents (`src/documents/`) - -`DocumentStorage` is the interface. There are two implementations: - -- `EventSourcedDocumentStorage(events, { projections?: KVNode })` is the v1 target. Projections default to the `~p` root on the same KV. -- `MongoDocumentStorage(client)` is a later milestone (section 9). - -```ts -const docs = new EventSourcedDocumentStorage(events, { indexes: { users: [{ fields: ['email'], unique: true }, { fields: ['age'] }] } }) -const users = docs.child('users') - -const id = await users.insert({ name: 'Ada', age: 36 }) // generates a time-sortable id -await users.set('123', { name: 'Ada', age: 36 }) -await users.update('123', { $set: { name: 'Ada L.' }, $inc: { logins: 1 } }) -await users.child('123').update({ $unset: { nickname: true } }) // self form -await users.get('123') -await users.delete('123') -for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) { -} -await users.findOne({ email: 'ada@x.io' }) -await users.history('123') // event-sourced only -await users.getAt('123', 17) // state as of version 17 -users.child('123').child('posts') // nested docs, same as KV -``` - -### 5.1 Events and projections - -- The doc at path `P` has its event stream at `P` in `EventStorage`. Its projection is at `~p/P` in the projections KV: `{ state, version }` (and `deleted: true` for tombstones). -- Event types: `{ op: 'set', doc }`, `{ op: 'patch', set?: Record, unset?: path[] }`, `{ op: 'delete' }`. -- **Store resolved deltas, not operators.** For example, `$inc` resolves to `{ set: { logins: 6 } }`, so replay is deterministic. -- Supported update operators in v1: `$set`, `$unset`, `$inc`, `$push`, `$pull`, `$addToSet`, with dot-path support. - -**Write path:** - -1. Read the projection. -2. Compute the new state and patch. -3. `append` with `expectedVersion = projection.version`. -4. Write the projection and index entries (atomic `batch` if available). -5. On `ConcurrencyError`: catch up the projection from the stream, recompute, and retry (bounded). - -**Reads** are a single projection `get`. Never replay on read. - -**Crash between append and projection write:** - -- The projection lags the stream. The next write's `expectedVersion` fails, which triggers catch-up, so the system self-heals. -- Provide `get(key, { consistent: true })` that checks the stream head and catches up first. -- Provide a `catchUp(node?)` maintenance method to repair a subtree. -- `insert` ids are time-sortable (ULID-like). Implement a small generator, with no dependency. - -### 5.2 Queries and document indexes - -- Filter subset: - - `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists` - - `$and`, `$or`, `$not` - - implicit equality and dot paths -- Options: `sort`, `limit`, `skip`, `projection` (fields). -- `find` on a node queries that node's **direct children**. -- Indexes are declared per path pattern (same trie mechanism as 3.3). - - Each index has `fields` (compound allowed); entries go under `~p//~i////…/`, using `encodeIndexValue`. - - `unique: true` additionally claims `~u//` via `setIfAbsent`. Throw at construction if the driver lacks `setIfAbsent`. -- **Planner:** - - Pick the index that covers the most leading equality fields, then at most one range field. - - Use the index order for `sort` when it matches. - - Otherwise do a full scan of children projections, with a one-time dev-mode `console.warn` naming the filter. - -## 6. Shared conventions - -- Every layer uses the same navigation (`child`, `as`, `keys`) and the same self/child arity rules. -- All multi-result reads are `AsyncIterable` (streamed, cancellable). Never build large arrays internally. -- Errors are typed classes exported from `src/errors.ts`: `ConcurrencyError`, `UniqueConstraintError`, `CapabilityError`, `InvalidKeyError`. -- Public exports go through `src/index.ts`, with subpath exports per layer if convenient. - -## 7. Types (`src/types/schema.ts`) - -Each node is typed by a single **Spec** that describes its own value and its children by key: - -```ts -export type Spec = { value?: unknown; children?: Record } -export type AnySpec = { value: unknown; children: Record } // untyped default - -export type ValueOf = S extends { value: infer V } ? V : never -export type ChildrenOf = S extends { children: infer C } ? C : {} -export type KeyOf = Extract, string | number> -export type ChildOf> = ChildrenOf[K] - -// Helpers -export type Many = {}> = { children: { [id: string]: { value: V; children: C } } } -export type Fields = { [K in keyof T & string]: { value: T[K] } } -``` - -Node signatures: - -```ts -class KVNode { - child>(key: K): KVNode> - get(): Promise | undefined> - get>(key: K): Promise> | undefined> - // set/delete/getMany/etc. follow the same pattern -} -``` - -Example to cover with `expectTypeOf` tests: - -```ts -type User = { username: string; age: number } -type Post = { title: string; createdAt: string } -type Schema = { - children: { - users: Many }> - profile: { children: { [id: string]: { children: Fields } } } // field-per-key style - } -} - -const db = new KVStorage(driver) -db.child('users').get('123') // User | undefined -db.child('users').child('123').get() // User | undefined (same key) -db.child('users').child('123').child('posts') // KVNode> -db.child('users').child('123').child('postz') // ✗ type error -db.child('profile').child('9').get('age') // number | undefined -db.child('profile').child('9').get('agee') // ✗ type error -``` - -Notes: - -- **Named levels** check keys at compile time; **dynamic `[id: string]` levels** accept any string. -- One node's children can't mix an index signature with named keys (a TS restriction), so schemas alternate between dynamic and named levels. That alternation exists only in types; the runtime API is just `child()`. -- `EventStorage` and `DocumentStorage` nodes use the same Spec machinery (the value type is the event type or the document type respectively). - -## 8. Performance requirements - -- Single-key ops: exactly one driver call (no hidden reads), except indexed writes (one `getMany` before the write) and document writes (projection read plus append plus projection write). -- `*Many` ops: one driver call each (plus index work), never N calls. -- `keys()`: skip-scan, O(direct children) seeks. -- Event head lookup: cached; cold lookup is a single reverse scan with `limit: 1`. -- Document `get`: one driver `get`. -- Add `vitest bench` benchmarks for: - - KV `get`/`set`/`getMany` at 10k/100k keys - - `keys()` on a node with 1k children each having 100 descendants - - event append and read - - document `get`/`update`/indexed `find` vs full-scan `find` -- Record baseline numbers in `DESIGN.md`. - -## 9. Milestones - -1. **Core:** `path.ts` (escape, unescape, encode, `succ`), `encodeIndexValue`, errors. Property tests for round-trips and order preservation. -2. **KV driver:** `KVDriver`, `BaseKVDriver`, `InMemoryKVDriver`, conformance suite. -3. **KV node:** full API from 3.2, including arity rules, skip-scan `keys`, `deleteTree`, `tree`. Add a model-based property test: random ops vs a reference `Map`. -4. **KV indexes:** pattern trie, atomic and non-atomic write paths, read-repair, range `findBy`. -5. **Events:** `KVEventStorage` with commit keys, `setIfAbsent` and mutex modes, head cache, binary-search `read({ from })`, concurrency tests (simulated interleaving with two storage instances on one driver). -6. **Documents:** `EventSourcedDocumentStorage`, update operators, projections, catch-up, queries, planner, unique indexes, `history` and `getAt`. -7. **Types:** Spec machinery wired through all three layers, plus `expectTypeOf` tests from section 7. -8. **HTTP driver:** - - A `createKVHttpHandler(driver)` (framework-agnostic `Request → Response`) plus an `HttpKVDriver` client. - - Endpoints: `POST /getMany`, `POST /setMany`, `POST /deleteMany`, `POST /batch`, `POST /setIfAbsent`, `GET /scan?gte<&limit&reverse&values` (NDJSON stream). - - It must pass the conformance suite end-to-end. -9. **Benchmarks** and a `DESIGN.md` update with results. -10. **(Later, ask first)** `MongoDocumentStorage`: - - Path pattern maps to a collection (e.g. `users/*/posts` → `users.posts`), with a `_parent` field and a compound index `(_parent, _id)`. - - Native indexes map from the same index declarations. - - It should pass the document test suite where semantics allow (no `history`/`getAt`). - -## 10. Decisions already made (don't revisit without asking) - -- `child()`-only navigation; no collection/doc classes. -- `get()` with no argument reads the node's **own key**. It never assembles the subtree; `tree()` does that explicitly. -- A node's own value and its children are independent keys. -- Self/child dispatch by arity; a passed-but-undefined key throws. -- Indexes are declared on path patterns at construction. -- Events use one key per commit, densely numbered. -- Documents store resolved deltas, and reads hit projections only. -- Keys sort lexicographically. -- Values are JSON-serialisable; byte drivers use a codec. - -## 11. Open questions to raise with the owner (don't block on them) - -- Order-preserving numeric **keys** (not just index values)? -- Should multi-writer safety be **required** (a hard error without `setIfAbsent`) rather than a documented single-writer fallback? -- The hard-delete story for immutable event logs (compaction or crypto-shredding) for privacy deletion requests. - -## 12. Implementation decisions - -Decisions taken where the brief was silent. Each is the simplest option consistent with sections 1–10. - -### M1 — Core - -- **Separator** must be a single printable ASCII character other than `%` and `~`. -- **Empty keys are rejected** (`InvalidKeyError`), as are `NaN`/`±Infinity` number keys. An empty segment - would produce `a//b`, which is ambiguous with the subtree layout. -- **Separator escaping** uses `%` + two uppercase hex digits (`/` → `%2F`, `%` → `%25`, leading `~` → `%7E`). -- **`InvalidKeyError` extends `TypeError`**, so "passed-but-undefined key throws a `TypeError`" (3.2) and - the typed error class (6) are the same throw. -- **Index string encoding deviates from "escaped like segments".** Percent-escaping is not - order-preserving (`"a/"` → `"a%2F"` sorts below `"a&"`, and `"a"` + sep sorts above `"a!"`), which - would break `findBy` range scans. Instead every character the index encoder emits sorts strictly - above the separator: code units `<= succ(sep)` are written as `succ(sep)` followed by - `chr(succ(sep) + 1 + c)`. This is monotone and prefix-safe, so `a < b ⇔ enc(a)+sep+… < enc(b)+sep+…` - exactly (property-tested). Encodings are still separator-free. -- **Type tags and number digits**: tags are the first five digits of the digit alphabet; numbers use - 16 digits of `0-9a-f` when the separator sorts below `'0'` (e.g. `/`), otherwise the 16 characters - just above the separator. `-0` is stored as `0`; `NaN` cannot be indexed. -- **Ordering** of strings is by UTF-16 code unit (JS `<`). Drivers that compare UTF-8 bytes agree - except for astral-plane characters vs. U+E000–U+FFFF; noted as a driver-author caveat. - -### M2 — KV driver - -- **Conformance suite location**: `src/testing/kv-conformance.ts`, exported as `potential-storage/testing`. - It imports `vitest`, which is an _optional peer dependency_; the core entry point never imports it. -- **`InMemoryKVDriver` clones values** on write and read (`structuredClone`, skipped for primitives) so - stored state can't be mutated through shared references, matching serialising drivers. Opt out - with `{ clone: false }`. -- **Scans tolerate concurrent writes**: the in-memory scan remembers the last key yielded and re-seeks - if the map changed while suspended. Read-repair and `deleteTree` rely on this. The conformance suite - requires it of every driver (deleting an already-yielded key mid-scan must not skip keys). -- **`setMany` bulk path**: ≥32 new keys are sorted and merged (O(n + k log k)); fewer are spliced. -- **`BaseKVDriver.deleteRange`** deletes in chunks of 1000 (scan with `limit`, then `deleteMany`, - repeat), so it never deletes from under its own live iterator. It also provides `get`/`set`/`delete` - single-key helpers. -- **No fake `batch`**: `BaseKVDriver` deliberately does not provide a non-atomic `batch`, since stores - use its presence to mean "atomic". - -### M3 — KV node - -- **Child order is "subtree order", not plain segment order.** The brief's skip-scan assumes a - child's keys are contiguous, but they aren't when a sibling extends a segment with a character - that sorts below the separator (for `/`: space and `!"#$%&'()*+,-.`). With children `a` and `a-1` - the key space is `p/a`, `p/a-1`, `p/a-1/x`, `p/a/x`, so `a`'s descendants sort _after_ `a-1`, and - "yield if different from the previous segment" would emit `a` twice. `keys()`/`entries()` therefore - order children by **escaped segment followed by the separator** (`a-1/` < `a/`), which is the order - of their subtrees in key space. For keys without such characters this is plain lexicographic order. - - Forward listing defers a child seen only through its own key on a small stack (bounded by - segment length) until the scan passes `p//`. It costs no extra driver calls beyond the - brief's one seek per child with descendants. - - Reverse listing needs one batched `getMany` probe per new child whose segment contains a - below-separator character, to find shorter siblings that only have an own key. - - `after` is exclusive in the direction of travel and follows the same order. - - Implemented once in `src/kv/children.ts` (`scanChildren`) and reused by the event and document - layers. A model-based property test checks forward, reverse, `after` and `limit` against - brute force with keys like `a`, `a-1`, `a.b`, `a/x`, `%`, `~t`. - - **Open question for the owner:** keep subtree order, or change segment escaping so the key - layout nests exactly. That would mean escaping every below-separator character with an escape - character above the separator, at the cost of the readable percent-encoding. -- **Errors from async methods are rejections.** Every async method (`get`, `set`, …) is an `async` - function, so a bad key yields a rejected promise rather than a synchronous throw. `child()` is sync - and throws synchronously. -- **`set(undefined)` is rejected** (`TypeError`: use `delete()`); `undefined` means "missing" in reads. -- **`deleteTree()` on the root** is allowed and clears the whole store (chunked scan + delete, since - the root range has no upper bound). Subtree deletion is `delete(selfKey)` plus - `deleteRange(prefix, selfKey + succ(sep))`. It is not `[selfKey, …)`, which would also catch - siblings such as `a!`. -- **`tree({ depth })`** builds a plain object: leaves are values, interior nodes are objects. A node - with both an own value and children exposes its own value under the `''` property (empty keys are - invalid, so it can't collide). Reserved subtrees are skipped. Deeper levels are skipped by seeking - rather than reading. -- **`keys()` and `entries()` yield string keys** (number keys come back as strings). -- **Exposed low-level properties** on every node: `path`, `selfKey`, `prefix`, `end` (subtree upper - bound), `driver`, `separator`, `isRoot`. The event and document layers build on these. - -### M4 — KV indexes - -- **Pattern syntax**: segments separated by the storage separator; `*` matches one segment; `''` - names the root. A literal `*` key can't be targeted by a pattern. -- **Overlapping patterns** that declare the same index name throw at construction. -- **Index trie state is stepped per `child()` call** (O(1) in patterns), so each node resolves its - indexes in O(depth) overall. A node holds two bound index sets: one over its children (for child - writes) and its parent's (for self-form writes such as `users.child('1').set(v)`). -- **Index function results**: `undefined` → not indexed; arrays are multi-entry and de-duplicated - per child; anything that is not `string | number | boolean | null` (or `NaN`) throws a `TypeError` - at write time. Index functions are not called for missing (`undefined`) values. -- **`findBy` results** are `[key, value]` pairs (the value is re-read anyway for verification). - Range queries accept `gt`/`gte`/`lt`/`lte`. Results are in index-value order, then child-key order, - and range queries de-duplicate multi-entry hits. -- **`findBy` batching**: entry keys are read in chunks of 128 (or of the remaining `limit`, if - smaller) and values fetched with one `getMany` per chunk. Stale entries are deleted with one - `deleteMany` per chunk. -- **Read-repair race (non-atomic drivers only)**: a reader can see a concurrent writer's fresh - index add before that writer's value write lands, judge it stale, and delete it. With multiple - writers on a driver without `batch`, this can lose an index entry. Mitigation: `readRepair: false` - on `KVStorage` (stale entries are then only skipped, never deleted). With `batch`, entries never go - stale, so the race doesn't arise. -- **`set` of an indexed child with duplicate keys in one `setMany`**: last write wins, and index - diffs are computed once per key. - -### M5 — Events - -- **`EventStorage` is an interface (type), not a class.** The brief's sample says `new EventStorage(kv)` - but also says it is the abstract interface; the concrete class is `KVEventStorage(kv, opts?)`, which - accepts a `KVStorage` or any `KVNode` as root. Options: `headCacheSize` (default 10 000), - `maxRetries` (default 5) and `now` (clock, default `Date.now`). -- **Commit value**: `{ firstVersion, at, events }`, with `at` in ms since epoch. Read results are - `{ version, commit, at, data }`. -- **The root node is not a stream** (consistent with the root having no own KV value). Non-root - nodes are both a stream and a container of child streams; `keys()` lists child streams, and a - node's `~e` subtree is reserved, so streams and children never mix. -- **`version()` always reads storage** (one reverse scan with `limit: 1`) and refreshes the head cache. - Only `append` trusts the cache. When a cached head disagrees with a numeric `expectedVersion`, - the head is re-read before throwing, so a stale cache never produces a false `ConcurrencyError`. -- **`append([])`** writes nothing; it still checks `expectedVersion` against a fresh head and returns it. -- **Read arity**: `read()` and `read(opts)` are the self form (the first argument is a non-null - object). Any other first argument is a key, so `read(undefined)` throws like every other - passed-but-undefined key. -- **Locating `from`/`to`**: commit `n` has `firstVersion >= n`, so the commit containing version `v` has - `seq <= v`. `read` probes `seq = v` first; with single-event commits that is the answer, in one - `getMany`. Otherwise it runs an 8-ary search, one `getMany` of up to 8 probes per round, which is - O(log₈ commits) round trips. The head is only read when the guess is past the end. -- **Mutex mode** (driver without `setIfAbsent`): a per-stream in-process promise chain serialises - appends. It is safe only with a single writer process. - -### M6 — Documents - -- **Projection value** is `{ state, version, commit }`, or `{ deleted: true, version, commit }` for a - tombstone. Tombstones keep the version, so re-creating a document continues its stream. -- **`expectedCommit` append hint.** Writes pass the projection's `commit` to `append` as - `expectedCommit`, so a write is exactly projection `getMany` + `setIfAbsent` + projection `batch` - (3 driver calls), even with a cold head cache. A hint taken from a projection can only be stale - (which causes a collision, then catch-up and retry), never ahead of the stream, so it cannot - create a gap. Non-KV backends may ignore it. -- **Per-document in-process write lock.** Writes to the same document within one storage instance - are serialised (`KeyedMutex`), so optimistic retries (default 5) are spent only on cross-process - contention. -- **Return values**: `set` → `void`; `insert` → id; `update` → new state, or `undefined` if the document - doesn't exist (pass `{ upsert: true }` to treat missing as `{}`); `delete` → whether a document - was deleted. No-op updates (resulting state equal to current) append nothing. -- **Arity for documents**: in forms with an optional leading key (`get`, `set`, `update`), a non-null - object first argument selects the self form, and any other first argument is a key (so `undefined` - throws). `delete`/`history` use argument count. `getAt(version)` vs `getAt(key, version)` also uses - argument count, because a version and a numeric key are both numbers. -- **The root document node has no own document** (consistent with KV). -- **`keys()` on a document node** lists children holding a live document. Unlike KV `keys()`, it skips - children that only have nested documents or tombstones. -- **`find` results** are `{ key, data }`. `findOne` returns the same shape or `undefined`. -- **Filter semantics vs MongoDB** (see `src/documents/filter.ts`): - - Comparisons are type-bracketed (`$gt: 5` matches only numbers), and cross-type order follows - the index encoding: `null/missing < false < true < numbers < strings < other`. - - On array fields, every positive operator in one condition (`$eq $gt $gte $lt $lte $in`) must be - satisfied by the **same element** (like `$elemMatch`). In MongoDB, `{ s: { $gt: 5, $lt: 10 } }` - matches `[1, 20]`; here it doesn't. This makes index range scans exact. To get MongoDB's - behaviour, split the condition with `$and`. - - `{ f: null }` matches missing fields. `$not` works both at top level (negating a filter) and at - field level. Unknown operators throw. - - `$in` is not index-accelerated in v1. -- **Index entries** for each field value: scalars as-is, arrays as their scalar elements (multikey; - empty arrays → `null`), and missing or non-scalar values as `null`. Every live document therefore - has at least one entry per index, so a full index scan sees all documents. Indexing a document - where more than one field of a compound index has multiple values throws (as MongoDB does). -- **Index names** default to the fields joined by `,`. -- **Unique indexes**: claims are scoped **per parent node** (`/~u//`), like one - MongoDB collection per parent. Value tuples where every field is missing/null are not claimed - (sparse). Claims are taken before appending, then released if the append fails. A claim value is - `[ownerKey, claimedAtMs]`. A claim held by another document is taken over only if that - document, after catching up from its stream, no longer holds the value, **and** the claim is older - than `claimGraceMs` (default 30 s). The owner might otherwise be between claiming and appending. - The residual risk is a writer stalling for longer than the grace period between claiming and - appending. -- **Planner**: per index, count leading equality fields (k), then whether field k has a range; - score = 2k + (range ? 1 : 0). Ties prefer an index whose order satisfies `sort` after dropping - equality-constrained sort fields. With no usable predicate, an index matching `sort` is still used - as an ordered full index scan, which streams and stops early with `limit`. Range bounds are clamped - to the bound's type class. Index-backed results are re-read from projections, re-checked against the - full filter, de-duplicated, and stale entries are read-repaired. -- **Result order** when `sort` is not given is the plan's order (index order, or child-key order for - a full scan). Ties in sorted results are broken by child key in the direction of the first sort - field, both for index-ordered and in-memory sorts. -- **In-memory sort** keeps a bounded top `skip + limit` buffer when a limit is given. Without a limit, - it buffers all matches; this is the one place results are materialised. -- **Full-scan warning**: `console.warn` once per filter _shape_ (values replaced by `?`) per storage - instance, only for non-empty filters. On by default unless `NODE_ENV === 'production'`; set it with - `warnOnFullScan`. -- **Catch-up** re-asserts unique claims for the caught-up state with plain writes, since the log is - the source of truth. A slow catch-up can briefly overwrite a newer projection with an older one; - the next write's `expectedVersion` check detects this and heals it. `catchUp()` walks the subtree - via event-stream `keys()` and returns the number of documents repaired. -- **Projections for non-KV event stores** must be passed explicitly (`projections: KVNode`); the - projection key of a document is then `projections.prefix + docKey`. - -### M7 — Types - -- **`AnySpec` uses an index signature** (`{ [key: string]: AnySpec }`), not `Record`, so - `KeyOf` is `string | number` and number keys type-check on untyped stores. -- **`ChildOf` guards with `infer C extends Spec`** so that `KVNode>` satisfies the `Spec` - constraint generically. -- **`setMany` entries are a union of precise `[key, value]` pairs** (`EntryOf`), so on named levels a - value must match its own key (`['age', 'x']` is rejected). -- **Listings**: `keys()` yields `` `${KeyOf}` `` (string literals on named levels, `string` on dynamic - ones). `entries()` and `findBy()` yield `[string, V]` on dynamic levels. -- **"Accept any node" parameters** are typed `KVNode` / `EventNode`. A typed node is - assignable to these, but not to `KVNode`, because `KeyOf` collapses to `never`. -- **Type tests** live in `test/types.test-d.ts` and are enforced by `tsc --noEmit` (part of `npm test`), - including the `@ts-expect-error` negative cases from section 7. - -### M8 — HTTP driver - -- **Extra endpoints** beyond the brief's list: `POST /deleteRange {gte, lt}` and - `GET /capabilities`. Unsupported optional operations return `501`, driver errors `500`, and unknown - routes `404`. -- **Missing vs `null`**: `/getMany` returns `{ values, missing }`, where `missing` lists the indices of - absent keys, because `null` is a legitimate stored value. -- **Codec scope**: the codec serialises whole request/response bodies and each NDJSON scan line - (`[key]` or `[key, value]`), so it must not emit newlines. The default is JSON. -- **Capabilities on the client**: `HttpKVDriver` defines `setIfAbsent`/`batch` only when the server - supports them, so stores' feature detection (e.g. unique indexes requiring `setIfAbsent`, atomic - index writes using `batch`) behaves as it would locally. `HttpKVDriver.connect()` discovers them via - `/capabilities`. The constructor takes them explicitly (default: all). Without native - `deleteRange` the client falls back to `BaseKVDriver`'s chunked delete. -- **Scan streaming**: the server pulls the driver iterator in chunks that grow from 16 to 256 - entries, so short reads (skip-scan re-seeks) don't make it run far ahead. Cancelling the client - iterator cancels the response body, which calls `return()` on the server-side iterator. -- **Optional `headers`** (e.g. auth) and an injectable `fetch` on the client; an optional `basePath` on - the handler. -- The conformance suite runs end-to-end in-process (fetch → handler), over a real `node:http` - socket, and against a minimal server driver with discovered capabilities. A smoke test runs KV - indexes, events and unique-indexed documents over HTTP. - -### M9 — Benchmarks and in-memory driver performance - -- **The in-memory driver's sorted array was replaced by `ChunkedSortedMap`** (sorted chunks of ≤1024 - keys, split in half on overflow; values in a `Map`). Profiling showed 59% of document-update time - in the sorted array's O(n) `splice` once the store held ~70k keys (each write adds commit, index - and claim keys). Inserts and deletes are now O(log n + 1024). Iteration is O(1) per step and - re-seeks after a mutation, which replaces the old version-counter revalidation in the driver. - The structure stays behind the `SortedMap` interface; a B-tree remains possible. -- **Value cloning uses a hand-written JSON deep copy** instead of `structuredClone` (about 10× cheaper - per value), in the in-memory driver and the document layer. Values are JSON by contract (§10). - - - -## 13. Benchmark baselines - -`npm run bench` (Vitest bench), in-memory driver with default cloning, Node 22.22, Linux x64 cloud -container (shared CPU, so expect ±10–20% run-to-run). Mean time per operation. - -**KV** (`bench/kv.bench.ts`): values are small objects. - -| Operation | 10k keys | 100k keys | -| -------------------- | -------- | --------- | -| `get` | 1.2 µs | 1.6 µs | -| `set` (overwrite) | 1.7 µs | 2.0 µs | -| `getMany` × 100 keys | 21 µs | 22 µs | - -**`keys()`** (`bench/keys.bench.ts`): node with 1 000 children, each with 100 descendants (101k keys). - -| Operation | Mean | -| -------------------------------- | ------ | -| `keys()` skip-scan (1 000 seeks) | 2.9 ms | -| Reference: full subtree scan | 36 ms | - -**Events** (`bench/events.bench.ts`): single-event commits. - -| Operation | Mean | -| ---------------------------------------------- | ------ | -| `append` 1 event (warm head cache) | 6.3 µs | -| `read` 100 events from version 5 000 of 10 000 | 174 µs | -| `read` full 10 000-event stream | 21 ms | -| `version()` (reverse scan, `limit: 1`) | 3.9 µs | - -**Documents** (`bench/documents.bench.ts`): 10 000 docs, indexes on `age` and unique `email`. - -| Operation | Mean | -| ----------------------------------------------------------------- | ------- | -| `get` | 2.6 µs | -| `update` (`$inc`, indexed node: projection read + append + batch) | 22 µs | -| `findOne` by unique index (`email`) | 16 µs | -| `find` indexed (`age = k`, ~100 hits) | 0.57 ms | -| `find` full scan (`city = k`, ~200 hits of 10k) | 28 ms | - -For comparison, the first run with the sorted-array map and `structuredClone` measured: KV `get` -3.7 µs, `getMany`×100 186 µs, `append` 34 µs, document `update` 110 µs, full scan 57 ms, and -full stream read 62 ms. +## 3. Keys (`utils/keys.ts`) + +- A node's key is its escaped path segments joined by `/` (`users/123`). The separator is fixed. +- **Escaping:** every character at or below `/` (space, `!"#$%&'()*+,-.`, `/`, control characters) + and `~` itself becomes `~XX` (uppercase hex). Escaped segments contain only characters above `/`, + so all keys of one child sort together (`p/a`, `p/a/…`, then `p/a~2D1` for `a-1`). That keeps + child listing a plain skip-scan. +- **Reserved segments** are `~` + a lowercase letter (`~e` = event commits). Escaping only produces + `~` + uppercase hex, so user keys can't collide. +- Keys sort lexicographically by escaped form (`10` before `9`; `a-1` → `a~2D1` after `a0`). +- Empty keys, `NaN`/`Infinity` and non-string/number keys are rejected. + +## 4. KV storage + +`BaseKVStorage` primitives (absolute keys). Implementations provide two reads and one protected +write; the public `writeMany(changes)` / `removeMany(keys)` live in the base class and emit a +`change` event after each write, so every store emits change events. + +| Primitive | | +| ------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| `readMany(keys)` | values in order, `undefined` for missing | +| `scan({ gte, lt, limit, reverse, values })` | key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once) | +| `write(changes)` → public `writeMany` | `{ key, value }` sets, `{ key }` deletes, in one batch | + +There is no conditional write. Stores execute exactly the operations they are given, and the last +write wins. Data that can be written concurrently belongs in event storage (section 5), where +concurrent appends never touch the same key. Data whose key is the hash of its content is +immutable, so concurrent writes of it are identical. + +**Values** are JSON plus `bigint` and `Uint8Array` (as in JSON, `-0` becomes `0` and `undefined` properties are dropped). Anything crossing the network is encoded as +Extended JSON (``, ``); IndexedDB stores them natively. + +**Events** (`EventEmitter`, shared by all nodes of a storage): `change` → +`{ changes: { key, value? }[], origin: 'local' | 'remote' }` (no `value` = removed), and +`resync` → changes may have been missed, so re-read. + +Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `set(key, value)`, +`setMany`, `setFields({ a, b })` / `getFields()` (direct children as an object), `delete(key?)`, +`deleteMany`, `deleteTree(key?)`, `keys({ limit, after, reverse })`, `entries(...)`. The root has +no own value. A node's own value and its children are independent keys. + +- **`keys()`** is a skip-scan: one seek per child that has descendants; siblings without + descendants share one iterator. It lists children that only have descendants, and skips + reserved segments. +- **The only KV "index" is the ordered key space.** There is no `find`: with KV you know where to look, + or you list a node's children. +- **Implementations:** + - `KVStorageInMemory`: `SortedMap` (sorted chunks of ≤1024 keys); values deep-copied. + - `KVStorageIndexedDB`: one object store with string keys. Scans read in batches, each in its own + transaction, resuming after the last key (IndexedDB transactions close when the code awaits + anything else). A write is one readwrite transaction. + - `KVStorageHttp` (client) and `KVStorageHttpServer` (wraps any KV storage; + `handle(Request) → Response`): `POST /readMany`, `/write`, `/scan` with Extended JSON bodies; + `scan` streams NDJSON. Unreachable server (fetch failure, 502/503/504) → + `StorageUnavailableError`. The server executes exactly what clients send and never reads values + for decisions, so values can be encrypted end to end. + - `KVStorageComposite`: see 4.2. `KVStorageEncrypted`: see 4.3. +- **CORS.** `allowOrigins` (`['https://app…']` or `'*'`; default none) answers preflights and adds + `Access-Control-Allow-Origin` for allowed origins. Allowed request headers: `content-type`, + `cache-control` (sent by `SSESession`), `last-event-id` and the auth headers. +- **Change stream (SSE).** `POST /events` with `{ paths }` streams every change to those keys and + their subtrees as `event: change` (data: the changes), including the subscriber's own writes. + `event: ready` follows each connection; keep-alive comments are sent every 15 s. The server keeps + no history: nothing is replayed after a reconnect. + - `KVStorageHttp.subscribe(path = this.key)` / `unsubscribe(path)` / `close()` manage the + subscribed paths. Each change reconnects the stream (vendored `SSESession`, backoff) with the + new list; `subscribe` resolves once connected. A 401/403 rejects instead of retrying. + - The client emits each pushed batch as `change` (`origin: 'remote'`), and `resync` whenever the + stream reconnects after a drop, since changes may have been missed meanwhile. + - One SSE connection per `KVStorageHttp` instance. Browsers allow 6 HTTP/1.1 connections per + host, so apps should share one client per server (or use HTTP/2); `close()` releases it. + - Limitation: a server only sees writes made through its own storage instance, so several server + processes over one database need a shared pub/sub (not built). +- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives. + Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be + encrypted freely. + +### 4.1 Authentication (`utils/request-auth.ts`) + +The server requires `auth`: either `{ verifiers, maxSkewMs?, now? }`, or `'none'` (open, for local +development and tests). It can't be left open by omission. + +- **Signed requests.** The client's `RequestSigner` signs `${timestamp}:${nonce}:${path}:${body}` + (nonce = 16 random bytes as hex, fresh per request; path = URL pathname; body = the exact request + body). Based on Harvmaster/sync-server-example, plus the nonce. Headers: `X-Public-Key` + (lowercase hex), `X-Timestamp` (ms), `X-Nonce`, `X-Signature` (hex) and `X-Signature-Algorithm`. + Every request is signed, and so is each SSE (re)connect, with a fresh timestamp and nonce. +- **Verification (401).** The server picks the `SignatureVerifier` registered for the algorithm + (unknown → 401), checks the nonce format (16–128 characters of `[0-9A-Za-z_-]`) and that the + timestamp is within `maxSkewMs` (default 60 s). It then rejects a nonce the same public key + already used within that window (replay protection), and verifies the signature over the + received timestamp, nonce, path and raw body. The body is parsed only afterwards. +- **Why a nonce.** Ed25519 signatures are deterministic, so without it two identical requests + from one key at the same millisecond (two tabs or devices subscribing to the same path, parallel + identical reads) would be indistinguishable, and the second rejected as a replay. +- **Algorithms are pluggable.** Built in: `Ed25519Signer` / `Ed25519Verifier` (WebCrypto, so no + dependencies; Node 22+ and current browsers; keys persist as PKCS #8). Others, e.g. secp256k1 via + libauth, implement `RequestSigner` (`algorithm`, `publicKey`, `sign`) and `SignatureVerifier` + (`algorithm`, `verify`). The server can accept several algorithms at once. +- **Authorization (403): the public key is the user id.** Every key in `readMany` and `write`, and + every subscription path, must be `` or under + `/`. A scan must stay within `[, 0)`, i.e. the key and its + subtree (escaping guarantees no other key falls in that range, including lookalikes such as + `x`). +- **Scoped events.** Each subscriber only receives changes under its own (authorized) paths. +- Proxies must preserve the URL path, because the signature covers it. + +### 4.2 `KVStorageComposite({ source, replicas, queue?, prefix?, retryMs? })` + +Keeps local copies of a source (e.g. `KVStorageHttp`) so reads are local and the app works +offline. With `prefix` (e.g. the user's public key) only that key and its subtree are synced, cached +and subscribed to; keys outside it throw `RangeError`. Without it, the whole source is copied, which +is only appropriate when the client may read all of it. + +- **Replicas** are listed fastest first (e.g. `[inMemory, indexedDb]`); the last one seeds the + others on start. Reads come from the first. +- **The queue** is any KV store (default: in memory). It holds writes the source couldn't take + (`writes/`) and whether a sync has completed (`synced`). Use a persistent node outside the + synced range, e.g. `indexedDb.child('~sync')` with `prefix: userKey`; a queue inside the range is + refused, since a sync would delete it. +- **Writes** are applied to every replica, then sent to the source, exactly as given. + - Applying locally first means the source relaying the write back finds nothing new. + - A write the source refuses (e.g. 403) is undone locally and rethrown. + - On `StorageUnavailableError`, or while a sync runs or writes are queued (to keep order), the + write goes to the queue and is sent later. The last write to reach the source wins. + - A queued write the source refuses on replay is dropped (retrying would block every later + write), the source's values for its keys are restored locally, and it is reported to `onError`. +- **Remote changes** pushed by the source are applied and announced (`change`, `origin: 'remote'`) + only where they differ from what the first replica holds. No write tagging or echo tracking is + needed: echoes of this client's writes change nothing. +- **`sync()`**: replay the queue in order (send, then re-apply locally) → merge-scan the source + against the last replica (both key-ordered), applying the difference in batches of 500, except + for keys with local writes the source hasn't confirmed yet (in flight or queued; counted per + key, and reloaded from a persistent queue on start), which would otherwise briefly revert → + replay again (writes made meanwhile) → mark `synced`. It runs on start, on `resync` (the stream + reconnected), and `retryMs` after a failure while offline. Called while a sync is running, it + schedules one more after it (shared by all callers meanwhile), so changes after the running + sync's scan aren't missed; e.g. the sync after subscribing, which closes the gap between the + first scan and the stream connecting. +- **Reads** wait for the first sync. Until one has completed, they go to the source. With a + persistent queue, a restart reads locally straight away (offline too). +- `start()` (also lazy on first use), `sync()`, `queued`, `close()`; `onError` receives errors no + caller can be rejected with (refused queued writes, failed syncs). Default `console.error`. +- Limitations: catching up re-scans the whole range (no server history); a remote change can + overwrite a value whose write is still queued until that write is replayed (last write wins); + replicas, source and queue must be root stores (the primitives use absolute keys). + +### 4.3 `KVStorageEncrypted(inner, { key })` + +Encrypts values before they reach another KV storage (usually `KVStorageHttp`), so the server only +stores ciphertext. + +- AES-GCM (WebCrypto) with a fresh 96-bit random IV per write. Stored value: `Uint8Array` + `[1 (format), iv (12), ciphertext + tag]`. The storage key is the additional authenticated data, + so a value moved to another key, or altered, fails to decrypt (`Error`, with the cause). +- Keys stay plaintext (the server needs them for authorization, ordering and prefix scans); value + sizes and timing are visible. +- Remote `change` events from the inner storage are decrypted (in order) and re-emitted; `resync` + and `subscribe` / `unsubscribe` pass through, so it works as a composite `source`. +- Keys: `generateKey()` (extractable), `exportKey` / `importKey` (raw), `deriveKey(password, salt, +iterations = 600 000)` (PBKDF2-SHA-256, not extractable). No key rotation yet. + +## 5. Event storage + +`BaseEventStorage` primitives over stream paths: `appendTo(stream, events)` → ids, +`readFrom(stream, { after, limit, reverse })`, `streamsOf`. Node API: `append(events)` / +`append(key, events)` → ids, `read(opts?)` / `read(key, opts?)` → `{ id, at, data }`, `keys()`. +The root is not a stream. + +**Appends never conflict.** Concurrent writers each add their own events, and every reader sees all +of them in the same order. There are no versions and no `expectedVersion`; folding events into +state is where concurrent changes meet (see event-sourced documents). + +**`EventStorageKV`** (on any KV storage): each event is one plain KV write, +`/~e/_` → the event. + +- Keys sort by time, so a stream reads in time order with one scan; `after` is an exclusive id. +- Events from different writers in the same millisecond order by hash: arbitrary, but the same for + every reader. One instance never reuses a time, so its own appends keep their order. +- The id can be recomputed from the content, and re-sending an event (e.g. a retried request) + writes the same key, so it can't be duplicated. +- An append is one `writeMany`, so it works through `KVStorageComposite` offline and syncs like any + other data. + +## 6. Document storage + +`BaseDocumentStorage` primitives over document paths: + +| Primitive | | +| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | +| `readMany(paths)` | | +| `mutate(path, change)` | maps current state → `DocChange` (or `null` = no-op), applies it, returns the new state; the implementation decides how it's made safe | +| `query(parent, DocumentQuery)` | children of `parent`, filtered, sorted and paged | + +Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)` → ULID-like id, +`update(spec, opts?)` / `update(key, spec, opts?)` → new state or `undefined` (`{ upsert }`), +`delete(key?)` → existed, `find(filter, { sort, limit, skip })`, `findOne`, `keys()`. + +- **Queries** (`utils/document-query.ts`): `{ field: value }` (dot paths; `null` also matches missing), + `$eq $gt $gte $lt $lte $in`. Comparisons are type-bracketed. On arrays a condition matches the array + or any one element. Sorting order: missing/null < booleans < numbers < strings < other. +- **Updates** (`utils/document-update.ts`): `$set $unset $inc $push $pull $addToSet` (`$each` for + push/addToSet). They resolve to a `DocChange` with final values (`$inc` → `set: { n: 6 }`), so + changes replay deterministically. +- **Indexes:** `{ indexes: { users: ['email', 'age'], 'users/*/posts': ['createdAt'] } }`, passed to + the implementation. + +**Implementations** + +- `DocumentStorageInMemory`: maps per collection; equality index per declared field (used for + `{ f: v }` / `{ f: { $eq: v } }`), otherwise scans. Writes are atomic in-process. +- `DocumentStorageKV`: each document is the KV value at its path; queries scan the collection's + children; no indexes; concurrent writes to one document are last-writer-wins. +- `EventSourcedDocumentStorage(events, projections)`: each document's `DocChange`s are appended to + its own stream. Current state lives in **any** `BaseDocumentStorage` used as the projection store, + which serves all reads, queries and indexes. + - A write reads the projection, resolves the change (`$inc` → `set: { n: 6 }`), appends it, and + applies it to the projection. + - Concurrent edits to different fields both survive; the same field ends with the later event's + value. (Increments from two devices don't add up: each resolves to a final value.) + - Events written elsewhere (another device, arriving through sync) are folded in by + `catchUp(key?)`, which rebuilds the projection from the whole stream. It also repairs a + projection left behind by a crash between append and projection write. + - Extras: `history(key?)`, `getAt(eventId)` / `getAt(key, eventId)`. Children keep these methods + in their types through interface declaration merging (types only). + +## 7. Changes from the original brief + +- Drivers + node classes → one abstract class per layer (client and server share an API). +- KV indexes (`findBy`, pattern trie, order-preserving value encoding, read-repair) removed. + Document indexes are declared as fields and handled by each store; unique indexes and the query + planner were removed. +- Queries: `$ne $nin $exists $and $or $not` and projections dropped. +- Separator fixed to `/`, and every character ≤ `/` is escaped so child keys stay contiguous. This + removed the subtree-order workaround in child listing. +- Event-sourced documents are events + a projection store instead of a bespoke projection format. +- No conditional writes and no versions: KV stores execute exactly what they're given (so a server + never needs to read values, which may be encrypted), and events get time + content-hash ids so + concurrent appends never collide. Optimistic concurrency (`expectedVersion`) was dropped. +- `tree()` removed (use `entries()` for field-per-key reads). + +## 8. Open questions + +- Composite: lazily syncing parts of a large subtree on first read; catching up without a full + re-scan (would need server-side history or a change log in event storage). +- Event-sourced documents: subscribing to event streams to run `catchUp` automatically. +- Authorization beyond "public key = user id" (sharing a subtree with other keys). +- Encryption: key rotation (re-encrypt, or a key id in the value format); sharing encrypted data + between users. +- Multi-process servers need a shared change feed (pub/sub) for SSE. + +- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The + encryption wrapper could support per-subject keys. +- Order-preserving numeric keys (`9` before `10`) as an opt-in key encoding. + +## 9. Benchmark baselines + +`npm run bench`, in-memory stores, Node 22, Linux x64 cloud container (shared CPU, ±10–20%). Mean per operation. + +| Benchmark | Mean | +| -------------------------------------------------------------------- | --------------- | +| KV `get` / `set` @ 10k keys | 1.5 / 1.7 µs | +| KV `get` / `set` @ 100k keys | 2.0 / 2.3 µs | +| KV `getMany` × 100 | 25–35 µs | +| `keys()` on 1k children × 100 descendants (skip-scan vs full scan) | 2.8 ms vs 34 ms | +| Event `append` (one SHA-256 via WebCrypto + one write) | 72 µs | +| Event `read` 100 from the middle of 10k | 119 µs | +| Event `read` full 10k stream | 11 ms | +| Event-sourced document `update ($inc)` | 86 µs | +| Documents in-memory @ 10k: `get` / `update` | 1.8 / 6.6 µs | +| Documents in-memory: `find` indexed (~100 hits) / scan (~200 of 10k) | 0.20 / 11 ms | +| Event-sourced @ 10k: `get` / `update` | 1.6 / 17 µs | +| Event-sourced: `find` indexed / scan | 0.38 / 12 ms | diff --git a/README.md b/README.md new file mode 100644 index 0000000..71d769c --- /dev/null +++ b/README.md @@ -0,0 +1,294 @@ +# @harvmaster/storage + +Layered TypeScript storage that works the same in the browser and on the server: + +- **KV storage**: ordered keys and values, navigated as a tree (`db.child('users').get('1')`). + In memory, IndexedDB, or over HTTP, with live updates over Server-Sent Events. +- **Event storage**: append-only streams on top of any KV storage. +- **Document storage**: JSON documents with queries, updates (`$set`, `$inc`, …) and indexes, on + top of any KV storage, or event-sourced. +- **Sync**: local replicas that work offline and stay up to date with a server + (`KVStorageComposite`), end-to-end encryption (`KVStorageEncrypted`) and signed requests where + the user's public key is their id. + +No runtime dependencies. Node 22+ and current browsers. Values are JSON plus `bigint` and +`Uint8Array`. + +> **Status: experimental (0.x).** The API may change between minor versions. See +> [DESIGN.md](./DESIGN.md) for how everything works and its known limitations. + +## Install + +The package is published to a private registry. Add to your project's `.npmrc`: + +```ini +@harvmaster:registry=https://verdaccio.harvmaster.com/ +``` + +```sh +npm install @harvmaster/storage # latest release +npm install @harvmaster/storage@next # `next` branch +npm install @harvmaster/storage@dev-sync # a development branch (`dev-`) +``` + +## KV storage + +Every store is a tree of nodes. `child(key)` returns a view rooted deeper (no I/O); methods take +an optional key to act on a child instead of the node itself. + +```ts +import { KVStorageInMemory } from '@harvmaster/storage' + +const db = new KVStorageInMemory() +const users = db.child('users') + +await users.set('1', { name: 'Ada' }) +await users.child('1').get() // { name: 'Ada' } +await users.get('1') // the same +await users.setMany([ + ['2', { name: 'Bob' }], + ['3', { name: 'Cy' }], +]) +await users.getMany(['1', 'missing']) // [{ name: 'Ada' }, undefined] + +for await (const key of users.keys({ limit: 10 })) console.log(key) // '1', '2', '3' +for await (const [key, user] of users.entries({ reverse: true })) console.log(key, user) + +await users.delete('3') +await users.deleteTree() // users and everything under it +``` + +A node's own value and its children are independent, so an object can also be stored field per key: + +```ts +const settings = db.child('settings') +await settings.setFields({ theme: 'dark', fontSize: 14 }) +await settings.get('theme') // 'dark' +await settings.getFields() // { fontSize: 14, theme: 'dark' } +``` + +Keys sort as strings (`'10'` before `'9'`), and any string or number is a valid key: special +characters are escaped. Every write emits a `change` event: + +```ts +db.on('change', ({ changes, origin }) => console.log(origin, changes)) // 'local', [{ key: 'users/1', value: … }] +``` + +### Types + +Describe the tree once to get typed keys and values: + +```ts +import { KVStorageInMemory, type Many, type Fields } from '@harvmaster/storage' + +type User = { name: string } +type Schema = { children: { users: Many; settings: { children: Fields<{ theme: string; fontSize: number }> } } } + +const typed = new KVStorageInMemory() +await typed.child('users').set('1', { name: 'Ada' }) // checked +const theme = await typed.child('settings').get('theme') // string | undefined +``` + +### IndexedDB + +```ts +import { KVStorageIndexedDB } from '@harvmaster/storage' + +const local = new KVStorageIndexedDB({ name: 'my-app' }) +``` + +## Event storage + +Append-only streams. Every node is a stream (its own events) and a parent of other streams. +Appends never conflict: concurrent writers each add their own events, and every reader sees all of +them in the same (time) order. + +```ts +import { EventStorageKV, KVStorageInMemory } from '@harvmaster/storage' + +const events = new EventStorageKV(new KVStorageInMemory()) +const orders = events.child('orders') + +const [id] = await orders.append('o1', [{ type: 'created', total: 30 }]) +await orders.child('o1').append([{ type: 'paid' }]) + +for await (const event of orders.read('o1')) console.log(event.id, event.at, event.data) +for await (const event of orders.read('o1', { after: id })) console.log(event.data) // { type: 'paid' } +for await (const key of orders.keys()) console.log(key) // 'o1' +``` + +Event ids are `