M1: core path escaping, order-preserving index value encoding, errors

Scaffold TypeScript/Vitest project, add DESIGN.md (brief) and CLAUDE.md. Property tests cover escape round-trips, separator-freedom and order preservation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
2026-10-23 06:52:06 +00:00
commit 69108d1557
14 changed files with 2665 additions and 0 deletions
@@ -0,0 +1,4 @@
 node_modules
 dist
 coverage
 *.log
@@ -0,0 +1,20 @@
 # PotentialStorage
 Layered TypeScript storage library: KV → Events → Documents. The full design, milestones and
 implementation decisions are in [DESIGN.md](./DESIGN.md) — read it before changing anything.
 Record new decisions in DESIGN.md section 12.
 ## Commands
 - `npm test` — run all tests (Vitest; includes `*.test-d.ts` type tests)
 - `npm run typecheck` — `tsc --noEmit` over src, test and bench
 - `npm run bench` — `vitest bench`
 - `npm run build` — emit `dist/`
 ## Rules
 - No runtime dependencies in `src/`. Dev dependencies only.
 - Drivers stay flat and string-keyed; all path/index/escaping logic lives in the node classes.
 - Self vs child dispatch is by `arguments.length`; a passed-but-`undefined`/`null` key throws.
 - Multi-result reads are `AsyncIterable`; don't build large arrays internally.
 - Keep tests green and commit per milestone. Ask the owner before pushing.
@@ -0,0 +1,387 @@
 # PotentialStorage — Design
 This document is the design brief agreed with the project owner (sections 0–11), followed by
 implementation decisions made where the brief was silent (section 12) and benchmark baselines
 (section 13).
 ## 0. Setup and workflow
 - Repository: `git@git.harvmaster.com:Harvmaster/PotentialStorage.git` (SSH; self-hosted Git, not GitHub). Clone it. If you can't reach it, stop and tell the owner rather than starting a fresh repo elsewhere.
 - Inspect what's already in the repo first. If it has tooling (package.json, tsconfig, test runner), keep it. If it's empty, set up:
  - TypeScript (`strict: true`, `noUncheckedIndexedAccess: true`), ESM, Node ≥ 20
  - Vitest for tests and `vitest bench` for benchmarks; `expectTypeOf` for type tests
  - Package name `potential-storage` unless one already exists
  - **No runtime dependencies in the core.** Dev dependencies are fine (e.g. `fast-check` for property tests).
 - Copy this brief into the repo as `DESIGN.md` and create a short `CLAUDE.md` pointing at it plus the build/test commands.
 - Work milestone by milestone (section 9). Commit after each milestone with a clear message, and keep tests green. Ask the owner before pushing.
 ## 1. Principles
 1. **Three layers**, each usable on its own: **KV** → **Event** (append-only, built on KV) → **Document** (Mongo-like, built on Events or native).
 2. **Drivers vs stores.** Backends implement a tiny, flat, string-keyed *driver*. All path, child, index, escaping and typing logic lives once in the user-facing *node* classes, so every backend gets identical behaviour.
 3. **One navigation primitive: `child(key)`.** There are no collection/document classes. Every node is a store over its direct children, and `child(key)` re-roots to that key. Navigation is synchronous and does no I/O.
 4. **No methods on data.** Reads return plain values.
 5. **Performance is a requirement.** Batch operations are the primitive; scans are ordered and streamed; nothing replays or scans more than it must (section 8).
 ## 2. Paths and key encoding (`src/core/path.ts`)
 - A node's path is `readonly string[]` of **unescaped** segments. Number keys are converted with `String()`.
 - The separator is a single character, configurable, default `/`.
 - **Escaping** (per segment): percent-encode `%`, the separator, and a **leading** `~`. Escaped segments never contain a raw separator. Write `escapeSegment` / `unescapeSegment` with round-trip property tests.
 - **Reserved segments** start with `~` (user segments can't, because a leading `~` is escaped):
  - `~i` index entries
  - `~u` unique-index claims
  - `~e` event commits
  - `~p` document projections root
 - Each node precomputes in its constructor:
  - `selfKey` = escaped segments joined by the separator (e.g. `users/123`)
  - `prefix` = `selfKey + sep` (`''` for the root)
  - No string work per call beyond `prefix + escapeSegment(key)`.
 - Keys are ordered **lexicographically** (so `10` sorts before `9`). Document this. An order-preserving numeric key encoding is a possible later option, not part of v1.
 Example layout:
 ```
 users/123                          → KV value of users/123
 users/123/posts/p1                 → KV value of a nested child
 users/~i/byEmail/sada@x.io/123     → index entry (value: empty / true)
 orders/o1/~e/000000000007          → event commit #7 of stream orders/o1
 ~p/users/123                       → document projection { state, version, commit }
 ```
 ## 3. Layer 1: KV
 ### 3.1 Driver interface (`src/kv/driver.ts`)
 ```ts
 export interface ScanOptions {
  gte?: string
  lt?: string
  limit?: number
  reverse?: boolean
  values?: boolean          // when false/omitted, yield value as undefined
 }
 export interface KVDriver {
  getMany(keys: string[]): Promise<(unknown | undefined)[]>       // same order as keys
  setMany(entries: [key: string, value: unknown][]): Promise<void>
  deleteMany(keys: string[]): Promise<void>
  scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]>  // MUST be key-ordered
  // Optional capabilities. Stores feature-detect them and fall back where possible.
  setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]>  // true = written
  batch?(ops: BatchOp[]): Promise<void>                            // atomic multi-op write
  deleteRange?(gte: string, lt: string): Promise<void>
 }
 export type BatchOp =
  | { type: 'set'; key: string; value: unknown }
  | { type: 'delete'; key: string }
 ```
 - Values are passed through as JSON-serialisable values. Drivers that need bytes (HTTP, Redis, files) serialise internally via a pluggable `Codec` (default JSON).
 - Provide an abstract `BaseKVDriver` with default implementations of `deleteRange` (scan + deleteMany) and helpers, so a minimal driver implements only the four required methods.
 - **`InMemoryKVDriver`**: a sorted structure (sorted array with binary search is fine for v1; keep it behind an interface so it can become a B-tree). It implements every optional capability: `setIfAbsent`, atomic `batch` and `deleteRange`.
 - Write a reusable **driver conformance suite** `runKVDriverConformance(name, makeDriver)` covering ordering, `gte`/`lt` bounds, reverse, limit, `values` on/off, `getMany` order with missing keys, and each optional capability when present.
 ### 3.2 `KVStorage` / `KVNode` (`src/kv/node.ts`)
 ```ts
 const db = new KVStorage(driver, { separator: '/', indexes: { ... } })   // db is the root KVNode
 const users = db.child('users')
 const user  = users.child('123')
 ```
 API of every `KVNode` (types in section 7):
 | Method | Meaning |
 |---|---|
 | `child(key)` | Sync. Returns a node at `path + key`. |
 | `as<S>()` | Type assertion for untyped use; returns the same node retyped. |
 | `get()` / `get(key)` | Own value at `selfKey` / value at child key. |
 | `getMany(keys)` | Batch child read, same order, `undefined` for missing. |
 | `set(value)` / `set(key, value)` | Write own value / child value. |
 | `setMany(entries)` | Batch child write. |
 | `delete()` / `delete(key)` | Delete own value / child value only (not descendants). |
 | `deleteMany(keys)` | Batch child delete. |
 | `deleteTree()` / `deleteTree(key)` | Delete own value **and all descendants** (uses `deleteRange` if available). |
 | `keys(opts?)` | `AsyncIterable<string>` of **direct** child keys (unescaped). Options: `limit`, `after`, `reverse`. |
 | `entries(opts?)` | `AsyncIterable<[key, value]>` of direct children that have an own value. |
 | `tree({ depth })` | Assemble an object from descendant keys up to `depth` (for field-per-key style). Bounded and explicit; never the default. |
 | `findBy(index, value \| { gte?, lt? }, opts?)` | Index lookup over direct children (section 3.3). |
 **Arity dispatch (important):**
 - Self vs child is decided by `arguments.length`, never by checking for `undefined`.
 - If a key argument is passed but is `undefined` or `null`, throw a `TypeError` (this prevents `users.get(maybeId)` silently reading self).
 - The root node (`path = []`) has no own value: `get()`, `set(v)` and `delete()` on root throw.
 - Implement one private `#resolve(args)` that every method uses.
 **Direct-children listing (`keys`)** uses a skip-scan:
 1. Scan `gte: prefix`, `lt: prefix + succ(sep)`.
 2. For each key, take the first segment after `prefix`. Skip reserved (`~…`) segments. Yield the segment (unescaped) if it differs from the previous one.
 3. When the next key is a descendant of the segment just yielded, re-seek the driver to `gte: prefix + escSeg + succ(sep)` (where `succ(c)` is the next char code) instead of iterating through the subtree. Keep using the same iterator while keys are siblings.
 Listing must be O(direct children × seek cost), not O(subtree). A child that has no own value but has descendants **is** listed.
 ### 3.3 Indexes (`src/kv/indexes.ts`)
 Indexes are declared on **path patterns** in the `KVStorage` constructor. A pattern names a parent node, and its index applies to that node's direct children's values:
 ```ts
 new KVStorage(driver, {
  indexes: {
    'users':         { byEmail: (u: User) => u.email },
    'users/*/posts': { byDate:  (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
  },
 })
 ```
 - An index function returns `IndexValue | IndexValue[] | undefined`, where `IndexValue = string | number | boolean | null`. An array means multi-entry; `undefined` means not indexed.
 - Compile patterns into a **segment trie** (`*` matches one segment). A node resolves its index set **once, at construction**, in O(depth).
 - Entry key: `<nodePrefix>~i/<indexName>/<encodedValue>/<escapedChildKey>` with a `true` value.
 - **`encodeIndexValue` must be order-preserving and separator-safe:**
  - Type tag first, so values order by type: `null` < `false` < `true` < numbers < strings.
  - Numbers use an order-preserving float64 encoding (flip the sign bit for positives and all bits for negatives, then fixed-width hex).
  - Strings are escaped like segments.
  - This enables `findBy(index, { gte, lt })` range scans.
 - **Writes to an indexed node:**
  1. Read the old value(s) (`getMany`) and compute the diff of index entries.
  2. If the driver has `batch`, write the value and the index adds/removes atomically.
  3. Otherwise, write index **adds first**, then the value, then index **removes**. Treat index entries as hints: `findBy` re-reads each value, re-checks the index function, and lazily deletes stale entries (read-repair).
 - Non-indexed nodes must **not** pay the read-before-write cost.
 ## 4. Layer 2: Events (`src/events/`)
 An append-only log per stream. A stream is identified by a node path. Same `child()` navigation as KV.
 ```ts
 const events = new EventStorage(kv)            // takes a KVStorage (or a KVNode as root)
 const orders = events.child('orders')
 await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 })
 await orders.child('o1').append([{ type: 'paid' }])          // self form
 for await (const e of orders.read('o1', { from: 1 })) { /* { version, commit, at, data } */ }
 await orders.version('o1')
 for await (const id of orders.keys()) { }                    // child streams
 orders.child('o1').child('shipments')
 ```
 - `EventStorage` is the abstract interface. `KVEventStorage` is the KV-backed implementation. Leave room for native backends (EventStoreDB, Kafka) implementing the same interface.
 - **Arity:** `append(events, opts?)` vs `append(key, events, opts?)`. Dispatch on `Array.isArray(args[0])`, plus the undefined-key rule from 3.2. `read`, `version` and `keys` follow the 3.2 rules.
 - **Versions:** an empty stream has version 0, and the first event has version 1. `expectedVersion: number | 'any'` (default `'any'`).
 - **Storage: one key per commit, densely numbered.**
  - Key: `<streamSelfKey>/~e/<commitSeq zero-padded to 12>`
  - Value: `{ firstVersion, at, events: E[] }`
  - The commit sequence (not the event version) is the key so that **any** stale writer collides with the existing key. Keying by version would let a stale writer create overlapping version ranges.
 - **Append:**
  1. Get the head `{ commitSeq, version }` (from the cache, or a reverse scan with `limit: 1`).
  2. If `expectedVersion` is a number and doesn't match the head, throw `ConcurrencyError`.
  3. Call `setIfAbsent(nextCommitKey)`.
  4. On `false` (collision): refresh the head. If `expectedVersion` was given, throw `ConcurrencyError`; otherwise retry up to 5 times.
  5. Without `setIfAbsent`, fall back to a per-stream in-process mutex. Document that this mode is single-writer only.
 - **Head cache:** a bounded LRU keyed by stream path. Update it on successful append; invalidate it on collision.
 - **`read({ from, to, reverse, limit })`** streams events with metadata. To start at version `from`, locate the containing commit by binary search over commit numbers using `getMany` probes (O(log commits) round trips), then forward-scan.
 - There is no update or delete. Keys inside `~e` are never rewritten.
 ## 5. Layer 3: Documents (`src/documents/`)
 `DocumentStorage` is the interface. There are two implementations:
 - `EventSourcedDocumentStorage(events, { projections?: KVNode })` is the v1 target. Projections default to the `~p` root on the same KV.
 - `MongoDocumentStorage(client)` is a later milestone (section 9).
 ```ts
 const docs  = new EventSourcedDocumentStorage(events, { indexes: { 'users': [{ fields: ['email'], unique: true }, { fields: ['age'] }] } })
 const users = docs.child('users')
 const id = await users.insert({ name: 'Ada', age: 36 })      // generates a time-sortable id
 await users.set('123', { name: 'Ada', age: 36 })
 await users.update('123', { $set: { name: 'Ada L.' }, $inc: { logins: 1 } })
 await users.child('123').update({ $unset: { nickname: true } })   // self form
 await users.get('123')
 await users.delete('123')
 for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) { }
 await users.findOne({ email: 'ada@x.io' })
 await users.history('123')        // event-sourced only
 await users.getAt('123', 17)      // state as of version 17
 users.child('123').child('posts') // nested docs, same as KV
 ```
 ### 5.1 Events and projections
 - The doc at path `P` has its event stream at `P` in `EventStorage`. Its projection is at `~p/P` in the projections KV: `{ state, version }` (and `deleted: true` for tombstones).
 - Event types: `{ op: 'set', doc }`, `{ op: 'patch', set?: Record<path, value>, unset?: path[] }`, `{ op: 'delete' }`.
 - **Store resolved deltas, not operators.** For example, `$inc` resolves to `{ set: { logins: 6 } }`, so replay is deterministic.
 - Supported update operators in v1: `$set`, `$unset`, `$inc`, `$push`, `$pull`, `$addToSet`, with dot-path support.
 **Write path:**
 1. Read the projection.
 2. Compute the new state and patch.
 3. `append` with `expectedVersion = projection.version`.
 4. Write the projection and index entries (atomic `batch` if available).
 5. On `ConcurrencyError`: catch up the projection from the stream, recompute, and retry (bounded).
 **Reads** are a single projection `get`. Never replay on read.
 **Crash between append and projection write:**
 - The projection lags the stream. The next write's `expectedVersion` fails, which triggers catch-up, so the system self-heals.
 - Provide `get(key, { consistent: true })` that checks the stream head and catches up first.
 - Provide a `catchUp(node?)` maintenance method to repair a subtree.
 - `insert` ids are time-sortable (ULID-like). Implement a small generator, with no dependency.
 ### 5.2 Queries and document indexes
 - Filter subset:
  - `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`
  - `$and`, `$or`, `$not`
  - implicit equality and dot paths
 - Options: `sort`, `limit`, `skip`, `projection` (fields).
 - `find` on a node queries that node's **direct children**.
 - Indexes are declared per path pattern (same trie mechanism as 3.3).
  - Each index has `fields` (compound allowed); entries go under `~p/<node>/~i/<name>/<enc(field1)>/<enc(field2)>/…/<childKey>`, using `encodeIndexValue`.
  - `unique: true` additionally claims `~u/<name>/<enc(values)>` via `setIfAbsent`. Throw at construction if the driver lacks `setIfAbsent`.
 - **Planner:**
  - Pick the index that covers the most leading equality fields, then at most one range field.
  - Use the index order for `sort` when it matches.
  - Otherwise do a full scan of children projections, with a one-time dev-mode `console.warn` naming the filter.
 ## 6. Shared conventions
 - Every layer uses the same navigation (`child`, `as`, `keys`) and the same self/child arity rules.
 - All multi-result reads are `AsyncIterable` (streamed, cancellable). Never build large arrays internally.
 - Errors are typed classes exported from `src/errors.ts`: `ConcurrencyError`, `UniqueConstraintError`, `CapabilityError`, `InvalidKeyError`.
 - Public exports go through `src/index.ts`, with subpath exports per layer if convenient.
 ## 7. Types (`src/types/schema.ts`)
 Each node is typed by a single **Spec** that describes its own value and its children by key:
 ```ts
 export type Spec = { value?: unknown; children?: Record<string, Spec> }
 export type AnySpec = { value: unknown; children: Record<string, AnySpec> }   // untyped default
 export type ValueOf<S>    = S extends { value: infer V } ? V : never
 export type ChildrenOf<S> = S extends { children: infer C } ? C : {}
 export type KeyOf<S>      = Extract<keyof ChildrenOf<S>, string | number>
 export type ChildOf<S, K extends KeyOf<S>> = ChildrenOf<S>[K]
 // Helpers
 export type Many<V, C extends Record<string, Spec> = {}> =
  { children: { [id: string]: { value: V; children: C } } }
 export type Fields<T> = { [K in keyof T & string]: { value: T[K] } }
 ```
 Node signatures:
 ```ts
 class KVNode<S extends Spec = AnySpec> {
  child<K extends KeyOf<S>>(key: K): KVNode<ChildOf<S, K>>
  get(): Promise<ValueOf<S> | undefined>
  get<K extends KeyOf<S>>(key: K): Promise<ValueOf<ChildOf<S, K>> | undefined>
  // set/delete/getMany/etc. follow the same pattern
 }
 ```
 Example to cover with `expectTypeOf` tests:
 ```ts
 type User = { username: string; age: number }
 type Post = { title: string; createdAt: string }
 type Schema = { children: {
  users:   Many<User, { posts: Many<Post> }>
  profile: { children: { [id: string]: { children: Fields<User> } } }   // field-per-key style
 } }
 const db = new KVStorage<Schema>(driver)
 db.child('users').get('123')                   // User | undefined
 db.child('users').child('123').get()           // User | undefined (same key)
 db.child('users').child('123').child('posts')  // KVNode<Many<Post>>
 db.child('users').child('123').child('postz')  // ✗ type error
 db.child('profile').child('9').get('age')      // number | undefined
 db.child('profile').child('9').get('agee')     // ✗ type error
 ```
 Notes:
 - **Named levels** check keys at compile time; **dynamic `[id: string]` levels** accept any string.
 - One node's children can't mix an index signature with named keys (a TS restriction), so schemas alternate between dynamic and named levels. That alternation exists only in types; the runtime API is just `child()`.
 - `EventStorage` and `DocumentStorage` nodes use the same Spec machinery (the value type is the event type or the document type respectively).
 ## 8. Performance requirements
 - Single-key ops: exactly one driver call (no hidden reads), except indexed writes (one `getMany` before the write) and document writes (projection read plus append plus projection write).
 - `*Many` ops: one driver call each (plus index work), never N calls.
 - `keys()`: skip-scan, O(direct children) seeks.
 - Event head lookup: cached; cold lookup is a single reverse scan with `limit: 1`.
 - Document `get`: one driver `get`.
 - Add `vitest bench` benchmarks for:
  - KV `get`/`set`/`getMany` at 10k/100k keys
  - `keys()` on a node with 1k children each having 100 descendants
  - event append and read
  - document `get`/`update`/indexed `find` vs full-scan `find`
 - Record baseline numbers in `DESIGN.md`.
 ## 9. Milestones
 1. **Core:** `path.ts` (escape, unescape, encode, `succ`), `encodeIndexValue`, errors. Property tests for round-trips and order preservation.
 2. **KV driver:** `KVDriver`, `BaseKVDriver`, `InMemoryKVDriver`, conformance suite.
 3. **KV node:** full API from 3.2, including arity rules, skip-scan `keys`, `deleteTree`, `tree`. Add a model-based property test: random ops vs a reference `Map`.
 4. **KV indexes:** pattern trie, atomic and non-atomic write paths, read-repair, range `findBy`.
 5. **Events:** `KVEventStorage` with commit keys, `setIfAbsent` and mutex modes, head cache, binary-search `read({ from })`, concurrency tests (simulated interleaving with two storage instances on one driver).
 6. **Documents:** `EventSourcedDocumentStorage`, update operators, projections, catch-up, queries, planner, unique indexes, `history` and `getAt`.
 7. **Types:** Spec machinery wired through all three layers, plus `expectTypeOf` tests from section 7.
 8. **HTTP driver:**
   - A `createKVHttpHandler(driver)` (framework-agnostic `Request → Response`) plus an `HttpKVDriver` client.
   - Endpoints: `POST /getMany`, `POST /setMany`, `POST /deleteMany`, `POST /batch`, `POST /setIfAbsent`, `GET /scan?gte&lt&limit&reverse&values` (NDJSON stream).
   - It must pass the conformance suite end-to-end.
 9. **Benchmarks** and a `DESIGN.md` update with results.
 10. **(Later, ask first)** `MongoDocumentStorage`:
    - Path pattern maps to a collection (e.g. `users/*/posts` → `users.posts`), with a `_parent` field and a compound index `(_parent, _id)`.
    - Native indexes map from the same index declarations.
    - It should pass the document test suite where semantics allow (no `history`/`getAt`).
 ## 10. Decisions already made (don't revisit without asking)
 - `child()`-only navigation; no collection/doc classes.
 - `get()` with no argument reads the node's **own key**. It never assembles the subtree; `tree()` does that explicitly.
 - A node's own value and its children are independent keys.
 - Self/child dispatch by arity; a passed-but-undefined key throws.
 - Indexes are declared on path patterns at construction.
 - Events use one key per commit, densely numbered.
 - Documents store resolved deltas, and reads hit projections only.
 - Keys sort lexicographically.
 - Values are JSON-serialisable; byte drivers use a codec.
 ## 11. Open questions to raise with the owner (don't block on them)
 - Order-preserving numeric **keys** (not just index values)?
 - Should multi-writer safety be **required** (a hard error without `setIfAbsent`) rather than a documented single-writer fallback?
 - The hard-delete story for immutable event logs (compaction or crypto-shredding) for privacy deletion requests.
 ## 12. Implementation decisions
 Decisions taken where the brief was silent. Each is the simplest option consistent with sections 1–10.
 ### M1 — Core
 - **Separator** must be a single printable ASCII character other than `%` and `~`.
 - **Empty keys are rejected** (`InvalidKeyError`), as are `NaN`/`±Infinity` number keys. An empty segment
  would produce `a//b`, which is ambiguous with the subtree layout.
 - **Separator escaping** uses `%` + two uppercase hex digits (`/` → `%2F`, `%` → `%25`, leading `~` → `%7E`).
 - **`InvalidKeyError` extends `TypeError`**, so "passed-but-undefined key throws a `TypeError`" (3.2) and
  the typed error class (6) are the same throw.
 - **Index string encoding deviates from "escaped like segments".** Percent-escaping is not
  order-preserving (`"a/"` → `"a%2F"` sorts below `"a&"`, and `"a"` + sep sorts above `"a!"`), which
  would break `findBy` range scans. Instead every character the index encoder emits sorts strictly
  above the separator: code units `<= succ(sep)` are written as `succ(sep)` followed by
  `chr(succ(sep) + 1 + c)`. This is monotone and prefix-safe, so `a < b ⇔ enc(a)+sep+… < enc(b)+sep+…`
  exactly (property-tested). Encodings are still separator-free.
 - **Type tags and number digits**: tags are the first five digits of the digit alphabet; numbers use
  16 digits of `0-9a-f` when the separator sorts below `'0'` (e.g. `/`), otherwise the 16 characters
  just above the separator. `-0` is stored as `0`; `NaN` cannot be indexed.
 - **Ordering** of strings is by UTF-16 code unit (JS `<`). Drivers that compare UTF-8 bytes agree
  except for astral-plane characters vs. U+E000–U+FFFF; noted as a driver-author caveat.
 <!-- decisions appended per milestone -->
 ## 13. Benchmark baselines
 _Filled in at milestone 9._
@@ -0,0 +1,49 @@
 {
  "name": "potential-storage",
  "version": "0.1.0",
  "description": "Layered TypeScript storage: KV -> Events -> Documents",
  "type": "module",
  "engines": {
    "node": ">=20"
  },
  "main": "./dist/index.js",
  "types": "./dist/index.d.ts",
  "exports": {
    ".": {
      "types": "./dist/index.d.ts",
      "import": "./dist/index.js"
    },
    "./kv": {
      "types": "./dist/kv/index.d.ts",
      "import": "./dist/kv/index.js"
    },
    "./events": {
      "types": "./dist/events/index.d.ts",
      "import": "./dist/events/index.js"
    },
    "./documents": {
      "types": "./dist/documents/index.d.ts",
      "import": "./dist/documents/index.js"
    },
    "./http": {
      "types": "./dist/http/index.d.ts",
      "import": "./dist/http/index.js"
    }
  },
  "files": [
    "dist"
  ],
  "scripts": {
    "build": "tsc -p tsconfig.build.json",
    "typecheck": "tsc --noEmit",
    "test": "vitest run",
    "test:watch": "vitest",
    "bench": "vitest bench --run"
  },
  "devDependencies": {
    "@types/node": "^22.20.5",
    "fast-check": "^4.10.2",
    "typescript": "^5.9.3",
    "vitest": "^3.2.7"
  }
 }
@@ -0,0 +1,177 @@
 import { DEFAULT_SEPARATOR, succ } from './path.js'
 export type IndexValue = string | number | boolean | null
 /**
 * Order-preserving, separator-safe encoding of index values.
 *
 * Every character the encoder emits sorts strictly above the separator. That makes encodings
 * prefix-safe inside keys: `enc(a) + sep + …` sorts below `enc(b)` whenever `enc(a)` is a proper
 * prefix of `enc(b)`, so `a < b ⇔ key(a, …) < key(b, …)` and range scans over index entries are exact.
 *
 * Layout: one type-tag character, then a body.
 * - Type tags order `null < false < true < number < string`.
 * - Numbers: IEEE-754 float64, big-endian; positive values have the sign bit flipped, negative
 *   values have all bits flipped; written as 16 fixed-width "hex" digits. `-0` is normalised to `0`;
 *   `NaN` is rejected.
 * - Strings: each code unit `c` with `c <= E` (where `E = succ(sep)`) is written as `E` followed by
 *   `chr(E + 1 + c)`; every other code unit is written as-is. This is monotone, so order is kept.
 *
 * The 16 digits are `0-9a-f` when the separator sorts below `'0'` (e.g. `/`), otherwise the 16
 * characters immediately after the separator.
 */
 export interface IndexValueCodec {
  readonly sep: string
  encode(value: IndexValue): string
  decode(encoded: string): IndexValue
 }
 const codecs = new Map<string, IndexValueCodec>()
 export function indexValueCodec(sep: string = DEFAULT_SEPARATOR): IndexValueCodec {
  let codec = codecs.get(sep)
  if (!codec) {
    codec = makeCodec(sep)
    codecs.set(sep, codec)
  }
  return codec
 }
 export function encodeIndexValue(value: IndexValue, sep: string = DEFAULT_SEPARATOR): string {
  return indexValueCodec(sep).encode(value)
 }
 export function decodeIndexValue(encoded: string, sep: string = DEFAULT_SEPARATOR): IndexValue {
  return indexValueCodec(sep).decode(encoded)
 }
 /** Encodes a compound value as `enc(v1) + sep + enc(v2) + …`. */
 export function encodeIndexValues(values: readonly IndexValue[], sep: string = DEFAULT_SEPARATOR): string {
  const codec = indexValueCodec(sep)
  let out = ''
  for (let i = 0; i < values.length; i++) {
    if (i > 0) out += sep
    out += codec.encode(values[i]!)
  }
  return out
 }
 export function isIndexValue(v: unknown): v is IndexValue {
  return v === null || typeof v === 'string' || typeof v === 'boolean' || (typeof v === 'number' && !Number.isNaN(v))
 }
 function makeCodec(sep: string): IndexValueCodec {
  const sepCode = sep.charCodeAt(0)
  const digits =
    sepCode < 0x30 ? '0123456789abcdef' : Array.from({ length: 16 }, (_, i) => String.fromCharCode(sepCode + 1 + i)).join('')
  const digitValue = new Map<string, number>([...digits].map((d, i) => [d, i]))
  const TAG_NULL = digits[0]!
  const TAG_FALSE = digits[1]!
  const TAG_TRUE = digits[2]!
  const TAG_NUMBER = digits[3]!
  const TAG_STRING = digits[4]!
  const esc = succ(sep)
  const escCode = esc.charCodeAt(0)
  const view = new DataView(new ArrayBuffer(8))
  function hex32(n: number): string {
    let out = ''
    for (let shift = 28; shift >= 0; shift -= 4) out += digits[(n >>> shift) & 0xf]
    return out
  }
  function encodeNumber(n: number): string {
    if (Number.isNaN(n)) throw new TypeError('NaN cannot be indexed')
    view.setFloat64(0, n === 0 ? 0 : n) // normalise -0
    let hi = view.getUint32(0)
    let lo = view.getUint32(4)
    if (hi & 0x80000000) {
      hi = ~hi >>> 0
      lo = ~lo >>> 0
    } else {
      hi = (hi ^ 0x80000000) >>> 0
    }
    return hex32(hi) + hex32(lo)
  }
  function decodeNumber(body: string): number {
    if (body.length !== 16) throw new TypeError(`Bad encoded number ${JSON.stringify(body)}`)
    let hi = 0
    let lo = 0
    for (let i = 0; i < 8; i++) hi = (hi << 4) | digitValue.get(body[i]!)!
    for (let i = 8; i < 16; i++) lo = (lo << 4) | digitValue.get(body[i]!)!
    hi >>>= 0
    lo >>>= 0
    if (hi & 0x80000000) {
      hi = (hi ^ 0x80000000) >>> 0
    } else {
      hi = ~hi >>> 0
      lo = ~lo >>> 0
    }
    view.setUint32(0, hi)
    view.setUint32(4, lo)
    return view.getFloat64(0)
  }
  function encodeString(s: string): string {
    let out = ''
    let start = 0
    for (let i = 0; i < s.length; i++) {
      const c = s.charCodeAt(i)
      if (c <= escCode) {
        out += s.slice(start, i) + esc + String.fromCharCode(escCode + 1 + c)
        start = i + 1
      }
    }
    return start === 0 ? s : out + s.slice(start)
  }
  function decodeString(body: string): string {
    if (body.indexOf(esc) === -1) return body
    let out = ''
    for (let i = 0; i < body.length; i++) {
      const c = body[i]!
      if (c === esc) {
        const next = body.charCodeAt(++i)
        out += String.fromCharCode(next - escCode - 1)
      } else out += c
    }
    return out
  }
  return {
    sep,
    encode(value) {
      if (value === null) return TAG_NULL
      switch (typeof value) {
        case 'boolean':
          return value ? TAG_TRUE : TAG_FALSE
        case 'number':
          return TAG_NUMBER + encodeNumber(value)
        case 'string':
          return TAG_STRING + encodeString(value)
        default:
          throw new TypeError(`Cannot index value of type ${typeof value}`)
      }
    },
    decode(encoded) {
      const tag = encoded[0]
      const body = encoded.slice(1)
      switch (tag) {
        case TAG_NULL:
          return null
        case TAG_FALSE:
          return false
        case TAG_TRUE:
          return true
        case TAG_NUMBER:
          return decodeNumber(body)
        case TAG_STRING:
          return decodeString(body)
        default:
          throw new TypeError(`Bad encoded index value ${JSON.stringify(encoded)}`)
      }
    },
  }
 }
@@ -0,0 +1,109 @@
 import { InvalidKeyError } from '../errors.js'
 export const DEFAULT_SEPARATOR = '/'
 /** Reserved segments. User segments can never start with `~` (a leading `~` is escaped). */
 export const RESERVED = {
  index: '~i',
  unique: '~u',
  events: '~e',
  projections: '~p',
 } as const
 export type Key = string | number
 /** The character immediately after `c` (by UTF-16 code unit). */
 export function succ(c: string): string {
  return String.fromCharCode(c.charCodeAt(0) + 1)
 }
 export function validateSeparator(sep: string): void {
  if (sep.length !== 1) throw new TypeError(`Separator must be a single character, got ${JSON.stringify(sep)}`)
  const code = sep.charCodeAt(0)
  if (code <= 0x20 || code >= 0x7f || sep === '%' || sep === '~') {
    throw new TypeError(`Separator must be printable ASCII other than "%" and "~", got ${JSON.stringify(sep)}`)
  }
 }
 function hex2(code: number): string {
  return '%' + code.toString(16).toUpperCase().padStart(2, '0')
 }
 /** Converts a user key to a segment string, rejecting `undefined`/`null`/empty. */
 export function toSegment(key: unknown): string {
  if (key === undefined || key === null) throw new InvalidKeyError(`Key must not be ${key}`)
  if (typeof key === 'number') {
    if (!Number.isFinite(key)) throw new InvalidKeyError(`Numeric key must be finite, got ${key}`)
    return String(key)
  }
  if (typeof key !== 'string') throw new InvalidKeyError(`Key must be a string or number, got ${typeof key}`)
  if (key === '') throw new InvalidKeyError('Key must not be empty')
  return key
 }
 /**
 * Escapes one path segment: percent-encodes `%`, the separator, and a leading `~`.
 * The result never contains a raw separator and never starts with `~`.
 */
 export function escapeSegment(segment: string, sep: string = DEFAULT_SEPARATOR): string {
  // Fast path: nothing to escape (the overwhelmingly common case).
  if (segment.indexOf('%') === -1 && segment.indexOf(sep) === -1 && segment.charCodeAt(0) !== 0x7e) return segment
  let out = ''
  for (let i = 0; i < segment.length; i++) {
    const c = segment[i]!
    if (c === '%' || c === sep || (i === 0 && c === '~')) out += hex2(c.charCodeAt(0))
    else out += c
  }
  return out
 }
 export function unescapeSegment(escaped: string): string {
  if (escaped.indexOf('%') === -1) return escaped
  return escaped.replace(/%([0-9A-Fa-f]{2})/g, (_, h: string) => String.fromCharCode(parseInt(h, 16)))
 }
 export function isReservedSegment(escaped: string): boolean {
  return escaped.charCodeAt(0) === 0x7e // '~'
 }
 /** Joins unescaped segments into an escaped key. */
 export function joinPath(path: readonly string[], sep: string = DEFAULT_SEPARATOR): string {
  let out = ''
  for (let i = 0; i < path.length; i++) {
    if (i > 0) out += sep
    out += escapeSegment(path[i]!, sep)
  }
  return out
 }
 /** Splits an escaped key into unescaped segments. */
 export function splitPath(key: string, sep: string = DEFAULT_SEPARATOR): string[] {
  if (key === '') return []
  return key.split(sep).map(unescapeSegment)
 }
 /**
 * Precomputed key strings for a node path.
 * - `selfKey`: escaped segments joined by the separator (`''` for the root)
 * - `prefix`: `selfKey + sep` (`''` for the root) — every descendant key starts with it
 * - `end`: exclusive upper bound of the subtree range (`selfKey + succ(sep)`; `undefined` for the root)
 */
 export interface PathKeys {
  readonly path: readonly string[]
  readonly sep: string
  readonly selfKey: string
  readonly prefix: string
  readonly end: string | undefined
 }
 export function pathKeys(path: readonly string[], sep: string): PathKeys {
  const selfKey = joinPath(path, sep)
  const root = path.length === 0
  return {
    path,
    sep,
    selfKey,
    prefix: root ? '' : selfKey + sep,
    end: root ? undefined : selfKey + succ(sep),
  }
 }
@@ -0,0 +1,47 @@
 /** Base class for all PotentialStorage errors. */
 export class PotentialStorageError extends Error {
  constructor(message: string) {
    super(message)
    this.name = new.target.name
  }
 }
 /** An optimistic-concurrency check failed (stale `expectedVersion` or a lost commit race). */
 export class ConcurrencyError extends PotentialStorageError {
  constructor(
    readonly stream: string,
    readonly expectedVersion: number | 'any',
    readonly actualVersion: number,
  ) {
    super(`Concurrency conflict on "${stream}": expected version ${expectedVersion}, actual ${actualVersion}`)
  }
 }
 /** A unique index already holds the given value for another key. */
 export class UniqueConstraintError extends PotentialStorageError {
  constructor(
    readonly index: string,
    readonly values: readonly unknown[],
    readonly existingKey?: string,
  ) {
    super(`Unique index "${index}" already contains ${JSON.stringify(values)}${existingKey === undefined ? '' : ` (key "${existingKey}")`}`)
  }
 }
 /** The driver lacks an optional capability required by the requested feature. */
 export class CapabilityError extends PotentialStorageError {
  constructor(readonly capability: string, detail?: string) {
    super(`Driver lacks capability "${capability}"${detail ? `: ${detail}` : ''}`)
  }
 }
 /**
 * A key argument is invalid: passed but `undefined`/`null`, empty, or otherwise unusable.
 * Extends `TypeError` so callers can catch either.
 */
 export class InvalidKeyError extends TypeError {
  constructor(message: string) {
    super(message)
    this.name = 'InvalidKeyError'
  }
 }
@@ -0,0 +1,3 @@
 export * from './errors.js'
 export * from './core/path.js'
 export * from './core/encoding.js'
@@ -0,0 +1,72 @@
 import { describe, expect, it } from 'vitest'
 import fc from 'fast-check'
 import { decodeIndexValue, encodeIndexValue, type IndexValue } from '../../src/core/encoding.js'
 const separators = fc.constantFrom('/', ':', '.', '|', '#', '!')
 const indexValue: fc.Arbitrary<IndexValue> = fc.oneof(
  fc.constant(null),
  fc.boolean(),
  fc.double({ noNaN: true }),
  fc.integer(),
  fc.string({ unit: 'binary' }),
  fc.string({ unit: fc.constantFrom('/', ':', '0', '1', 'a', '%', '~', '\0', '!') }),
 )
 const rank = (v: IndexValue) => (v === null ? 0 : v === false ? 1 : v === true ? 2 : typeof v === 'number' ? 3 : 4)
 function compare(a: IndexValue, b: IndexValue): number {
  const ra = rank(a)
  const rb = rank(b)
  if (ra !== rb) return ra - rb
  if (typeof a === 'number' && typeof b === 'number') return a === b ? 0 : a < b ? -1 : 1
  if (typeof a === 'string' && typeof b === 'string') return a === b ? 0 : a < b ? -1 : 1
  return 0
 }
 const sign = (n: number) => (n < 0 ? -1 : n > 0 ? 1 : 0)
 const cmpStr = (a: string, b: string) => (a < b ? -1 : a > b ? 1 : 0)
 describe('encodeIndexValue', () => {
  it('round-trips (property)', () => {
    fc.assert(
      fc.property(indexValue, separators, (v, sep) => {
        const decoded = decodeIndexValue(encodeIndexValue(v, sep), sep)
        if (typeof v === 'number' && v === 0) expect(decoded).toBe(0) // -0 normalised
        else expect(decoded).toEqual(v)
      }),
    )
  })
  it('emits only characters above the separator (property)', () => {
    fc.assert(
      fc.property(indexValue, separators, (v, sep) => {
        for (const c of encodeIndexValue(v, sep)) expect(c > sep).toBe(true)
      }),
    )
  })
  it('preserves order, including when followed by a separator and suffix (property)', () => {
    fc.assert(
      fc.property(indexValue, indexValue, fc.string(), fc.string(), separators, (a, b, sa, sb, sep) => {
        const ea = encodeIndexValue(a, sep)
        const eb = encodeIndexValue(b, sep)
        const expected = sign(compare(a, b))
        expect(cmpStr(ea, eb)).toBe(expected)
        if (expected !== 0) expect(cmpStr(ea + sep + sa, eb + sep + sb)).toBe(expected)
      }),
      { numRuns: 2000 },
    )
  })
  it('orders types null < false < true < numbers < strings', () => {
    const values: IndexValue[] = ['', 'a', -Infinity, -1, 0, 1e300, Infinity, true, false, null]
    const sorted = [...values].sort((a, b) => cmpStr(encodeIndexValue(a), encodeIndexValue(b)))
    expect(sorted).toEqual([null, false, true, -Infinity, -1, 0, 1e300, Infinity, '', 'a'])
  })
  it('rejects NaN and non-index values', () => {
    expect(() => encodeIndexValue(NaN)).toThrow()
    expect(() => encodeIndexValue({} as never)).toThrow()
  })
 })
@@ -0,0 +1,82 @@
 import { describe, expect, it } from 'vitest'
 import fc from 'fast-check'
 import {
  escapeSegment,
  joinPath,
  pathKeys,
  splitPath,
  succ,
  toSegment,
  unescapeSegment,
  validateSeparator,
 } from '../../src/core/path.js'
 import { InvalidKeyError } from '../../src/errors.js'
 const separators = fc.constantFrom('/', ':', '.', '|', '#', '\\', '!')
 describe('escapeSegment', () => {
  it('escapes %, the separator and a leading ~', () => {
    expect(escapeSegment('a/b')).toBe('a%2Fb')
    expect(escapeSegment('50%')).toBe('50%25')
    expect(escapeSegment('~x')).toBe('%7Ex')
    expect(escapeSegment('x~')).toBe('x~')
    expect(escapeSegment('a:b', ':')).toBe('a%3Ab')
    expect(escapeSegment('a/b', ':')).toBe('a/b')
    expect(escapeSegment('plain')).toBe('plain')
  })
  it('round-trips (property)', () => {
    fc.assert(
      fc.property(fc.string({ unit: 'binary' }), separators, (s, sep) => {
        expect(unescapeSegment(escapeSegment(s, sep))).toBe(s)
      }),
    )
  })
  it('never emits a raw separator or a leading ~ (property)', () => {
    fc.assert(
      fc.property(fc.string({ unit: 'binary' }), separators, (s, sep) => {
        const e = escapeSegment(s, sep)
        expect(e.includes(sep)).toBe(false)
        expect(e.startsWith('~')).toBe(false)
      }),
    )
  })
  it('joins and splits paths (property)', () => {
    fc.assert(
      fc.property(fc.array(fc.string({ minLength: 1 }), { maxLength: 6 }), separators, (path, sep) => {
        expect(splitPath(joinPath(path, sep), sep)).toEqual(path)
      }),
    )
  })
 })
 describe('pathKeys', () => {
  it('precomputes selfKey, prefix and subtree end', () => {
    expect(pathKeys(['users', '123'], '/')).toMatchObject({ selfKey: 'users/123', prefix: 'users/123/', end: 'users/1230' })
    expect(pathKeys([], '/')).toMatchObject({ selfKey: '', prefix: '', end: undefined })
    expect(pathKeys(['a/b'], '/').selfKey).toBe('a%2Fb')
  })
 })
 describe('misc', () => {
  it('succ returns the next char', () => {
    expect(succ('/')).toBe('0')
  })
  it('toSegment rejects undefined, null, empty and non-finite', () => {
    expect(() => toSegment(undefined)).toThrow(InvalidKeyError)
    expect(() => toSegment(null)).toThrow(TypeError)
    expect(() => toSegment('')).toThrow(InvalidKeyError)
    expect(() => toSegment(NaN)).toThrow(InvalidKeyError)
    expect(toSegment(12)).toBe('12')
  })
  it('validates separators', () => {
    expect(() => validateSeparator('//')).toThrow()
    expect(() => validateSeparator('%')).toThrow()
    expect(() => validateSeparator('~')).toThrow()
    expect(() => validateSeparator(':')).not.toThrow()
  })
 })
@@ -0,0 +1,5 @@
 {
  "extends": "./tsconfig.json",
  "compilerOptions": { "rootDir": "src", "outDir": "dist" },
  "include": ["src"]
 }
@@ -0,0 +1,21 @@
 {
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "lib": ["ES2022"],
    "types": ["node"],
    "strict": true,
    "noUncheckedIndexedAccess": true,
    "noImplicitOverride": true,
    "exactOptionalPropertyTypes": false,
    "declaration": true,
    "sourceMap": true,
    "outDir": "dist",
    "rootDir": ".",
    "skipLibCheck": true,
    "isolatedModules": true,
    "verbatimModuleSyntax": true
  },
  "include": ["src", "test", "bench"]
 }
@@ -0,0 +1,9 @@
 import { defineConfig } from 'vitest/config'
 export default defineConfig({
  test: {
    include: ['test/**/*.test.ts'],
    typecheck: { enabled: true, include: ['test/**/*.test-d.ts'] },
  },
  bench: { include: ['bench/**/*.bench.ts'] },
 })