# PotentialStorage — Design

This document is the design brief agreed with the project owner (sections 0–11), followed by
implementation decisions made where the brief was silent (section 12) and benchmark baselines
(section 13).

## 0. Setup and workflow

- Repository: `git@git.harvmaster.com:Harvmaster/PotentialStorage.git` (SSH; self-hosted Git, not GitHub). Clone it. If you can't reach it, stop and tell the owner rather than starting a fresh repo elsewhere.
- Inspect what's already in the repo first. If it has tooling (package.json, tsconfig, test runner), keep it. If it's empty, set up:
  - TypeScript (`strict: true`, `noUncheckedIndexedAccess: true`), ESM, Node ≥ 20
  - Vitest for tests and `vitest bench` for benchmarks; `expectTypeOf` for type tests
  - Package name `potential-storage` unless one already exists
  - **No runtime dependencies in the core.** Dev dependencies are fine (e.g. `fast-check` for property tests).
- Copy this brief into the repo as `DESIGN.md` and create a short `CLAUDE.md` pointing at it plus the build/test commands.
- Work milestone by milestone (section 9). Commit after each milestone with a clear message, and keep tests green. Ask the owner before pushing.

## 1. Principles

1. **Three layers**, each usable on its own: **KV** → **Event** (append-only, built on KV) → **Document** (Mongo-like, built on Events or native).
2. **Drivers vs stores.** Backends implement a tiny, flat, string-keyed *driver*. All path, child, index, escaping and typing logic lives once in the user-facing *node* classes, so every backend gets identical behaviour.
3. **One navigation primitive: `child(key)`.** There are no collection/document classes. Every node is a store over its direct children, and `child(key)` re-roots to that key. Navigation is synchronous and does no I/O.
4. **No methods on data.** Reads return plain values.
5. **Performance is a requirement.** Batch operations are the primitive; scans are ordered and streamed; nothing replays or scans more than it must (section 8).

## 2. Paths and key encoding (`src/core/path.ts`)

- A node's path is `readonly string[]` of **unescaped** segments. Number keys are converted with `String()`.
- The separator is a single character, configurable, default `/`.
- **Escaping** (per segment): percent-encode `%`, the separator, and a **leading** `~`. Escaped segments never contain a raw separator. Write `escapeSegment` / `unescapeSegment` with round-trip property tests.
- **Reserved segments** start with `~` (user segments can't, because a leading `~` is escaped):
  - `~i` index entries
  - `~u` unique-index claims
  - `~e` event commits
  - `~p` document projections root
- Each node precomputes in its constructor:
  - `selfKey` = escaped segments joined by the separator (e.g. `users/123`)
  - `prefix` = `selfKey + sep` (`''` for the root)
  - No string work per call beyond `prefix + escapeSegment(key)`.
- Keys are ordered **lexicographically** (so `10` sorts before `9`). Document this. An order-preserving numeric key encoding is a possible later option, not part of v1.

Example layout:

```
users/123                          → KV value of users/123
users/123/posts/p1                 → KV value of a nested child
users/~i/byEmail/sada@x.io/123     → index entry (value: empty / true)
orders/o1/~e/000000000007          → event commit #7 of stream orders/o1
~p/users/123                       → document projection { state, version, commit }
```

## 3. Layer 1: KV

### 3.1 Driver interface (`src/kv/driver.ts`)

```ts
export interface ScanOptions {
  gte?: string
  lt?: string
  limit?: number
  reverse?: boolean
  values?: boolean          // when false/omitted, yield value as undefined
}

export interface KVDriver {
  getMany(keys: string[]): Promise<(unknown | undefined)[]>       // same order as keys
  setMany(entries: [key: string, value: unknown][]): Promise<void>
  deleteMany(keys: string[]): Promise<void>
  scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]>  // MUST be key-ordered

  // Optional capabilities. Stores feature-detect them and fall back where possible.
  setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]>  // true = written
  batch?(ops: BatchOp[]): Promise<void>                            // atomic multi-op write
  deleteRange?(gte: string, lt: string): Promise<void>
}

export type BatchOp =
  | { type: 'set'; key: string; value: unknown }
  | { type: 'delete'; key: string }
```

- Values are passed through as JSON-serialisable values. Drivers that need bytes (HTTP, Redis, files) serialise internally via a pluggable `Codec` (default JSON).
- Provide an abstract `BaseKVDriver` with default implementations of `deleteRange` (scan + deleteMany) and helpers, so a minimal driver implements only the four required methods.
- **`InMemoryKVDriver`**: a sorted structure (sorted array with binary search is fine for v1; keep it behind an interface so it can become a B-tree). It implements every optional capability: `setIfAbsent`, atomic `batch` and `deleteRange`.
- Write a reusable **driver conformance suite** `runKVDriverConformance(name, makeDriver)` covering ordering, `gte`/`lt` bounds, reverse, limit, `values` on/off, `getMany` order with missing keys, and each optional capability when present.

### 3.2 `KVStorage` / `KVNode` (`src/kv/node.ts`)

```ts
const db = new KVStorage(driver, { separator: '/', indexes: { ... } })   // db is the root KVNode
const users = db.child('users')
const user  = users.child('123')
```

API of every `KVNode` (types in section 7):

| Method | Meaning |
|---|---|
| `child(key)` | Sync. Returns a node at `path + key`. |
| `as<S>()` | Type assertion for untyped use; returns the same node retyped. |
| `get()` / `get(key)` | Own value at `selfKey` / value at child key. |
| `getMany(keys)` | Batch child read, same order, `undefined` for missing. |
| `set(value)` / `set(key, value)` | Write own value / child value. |
| `setMany(entries)` | Batch child write. |
| `delete()` / `delete(key)` | Delete own value / child value only (not descendants). |
| `deleteMany(keys)` | Batch child delete. |
| `deleteTree()` / `deleteTree(key)` | Delete own value **and all descendants** (uses `deleteRange` if available). |
| `keys(opts?)` | `AsyncIterable<string>` of **direct** child keys (unescaped). Options: `limit`, `after`, `reverse`. |
| `entries(opts?)` | `AsyncIterable<[key, value]>` of direct children that have an own value. |
| `tree({ depth })` | Assemble an object from descendant keys up to `depth` (for field-per-key style). Bounded and explicit; never the default. |
| `findBy(index, value \| { gte?, lt? }, opts?)` | Index lookup over direct children (section 3.3). |

**Arity dispatch (important):**
- Self vs child is decided by `arguments.length`, never by checking for `undefined`.
- If a key argument is passed but is `undefined` or `null`, throw a `TypeError` (this prevents `users.get(maybeId)` silently reading self).
- The root node (`path = []`) has no own value: `get()`, `set(v)` and `delete()` on root throw.
- Implement one private `#resolve(args)` that every method uses.

**Direct-children listing (`keys`)** uses a skip-scan:
1. Scan `gte: prefix`, `lt: prefix + succ(sep)`.
2. For each key, take the first segment after `prefix`. Skip reserved (`~…`) segments. Yield the segment (unescaped) if it differs from the previous one.
3. When the next key is a descendant of the segment just yielded, re-seek the driver to `gte: prefix + escSeg + succ(sep)` (where `succ(c)` is the next char code) instead of iterating through the subtree. Keep using the same iterator while keys are siblings.

Listing must be O(direct children × seek cost), not O(subtree). A child that has no own value but has descendants **is** listed.

### 3.3 Indexes (`src/kv/indexes.ts`)

Indexes are declared on **path patterns** in the `KVStorage` constructor. A pattern names a parent node, and its index applies to that node's direct children's values:

```ts
new KVStorage(driver, {
  indexes: {
    'users':         { byEmail: (u: User) => u.email },
    'users/*/posts': { byDate:  (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
  },
})
```

- An index function returns `IndexValue | IndexValue[] | undefined`, where `IndexValue = string | number | boolean | null`. An array means multi-entry; `undefined` means not indexed.
- Compile patterns into a **segment trie** (`*` matches one segment). A node resolves its index set **once, at construction**, in O(depth).
- Entry key: `<nodePrefix>~i/<indexName>/<encodedValue>/<escapedChildKey>` with a `true` value.
- **`encodeIndexValue` must be order-preserving and separator-safe:**
  - Type tag first, so values order by type: `null` < `false` < `true` < numbers < strings.
  - Numbers use an order-preserving float64 encoding (flip the sign bit for positives and all bits for negatives, then fixed-width hex).
  - Strings are escaped like segments.
  - This enables `findBy(index, { gte, lt })` range scans.
- **Writes to an indexed node:**
  1. Read the old value(s) (`getMany`) and compute the diff of index entries.
  2. If the driver has `batch`, write the value and the index adds/removes atomically.
  3. Otherwise, write index **adds first**, then the value, then index **removes**. Treat index entries as hints: `findBy` re-reads each value, re-checks the index function, and lazily deletes stale entries (read-repair).
- Non-indexed nodes must **not** pay the read-before-write cost.

## 4. Layer 2: Events (`src/events/`)

An append-only log per stream. A stream is identified by a node path. Same `child()` navigation as KV.

```ts
const events = new EventStorage(kv)            // takes a KVStorage (or a KVNode as root)
const orders = events.child('orders')

await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 })
await orders.child('o1').append([{ type: 'paid' }])          // self form
for await (const e of orders.read('o1', { from: 1 })) { /* { version, commit, at, data } */ }
await orders.version('o1')
for await (const id of orders.keys()) { }                    // child streams
orders.child('o1').child('shipments')
```

- `EventStorage` is the abstract interface. `KVEventStorage` is the KV-backed implementation. Leave room for native backends (EventStoreDB, Kafka) implementing the same interface.
- **Arity:** `append(events, opts?)` vs `append(key, events, opts?)`. Dispatch on `Array.isArray(args[0])`, plus the undefined-key rule from 3.2. `read`, `version` and `keys` follow the 3.2 rules.
- **Versions:** an empty stream has version 0, and the first event has version 1. `expectedVersion: number | 'any'` (default `'any'`).
- **Storage: one key per commit, densely numbered.**
  - Key: `<streamSelfKey>/~e/<commitSeq zero-padded to 12>`
  - Value: `{ firstVersion, at, events: E[] }`
  - The commit sequence (not the event version) is the key so that **any** stale writer collides with the existing key. Keying by version would let a stale writer create overlapping version ranges.
- **Append:**
  1. Get the head `{ commitSeq, version }` (from the cache, or a reverse scan with `limit: 1`).
  2. If `expectedVersion` is a number and doesn't match the head, throw `ConcurrencyError`.
  3. Call `setIfAbsent(nextCommitKey)`.
  4. On `false` (collision): refresh the head. If `expectedVersion` was given, throw `ConcurrencyError`; otherwise retry up to 5 times.
  5. Without `setIfAbsent`, fall back to a per-stream in-process mutex. Document that this mode is single-writer only.
- **Head cache:** a bounded LRU keyed by stream path. Update it on successful append; invalidate it on collision.
- **`read({ from, to, reverse, limit })`** streams events with metadata. To start at version `from`, locate the containing commit by binary search over commit numbers using `getMany` probes (O(log commits) round trips), then forward-scan.
- There is no update or delete. Keys inside `~e` are never rewritten.

## 5. Layer 3: Documents (`src/documents/`)

`DocumentStorage` is the interface. There are two implementations:
- `EventSourcedDocumentStorage(events, { projections?: KVNode })` is the v1 target. Projections default to the `~p` root on the same KV.
- `MongoDocumentStorage(client)` is a later milestone (section 9).

```ts
const docs  = new EventSourcedDocumentStorage(events, { indexes: { 'users': [{ fields: ['email'], unique: true }, { fields: ['age'] }] } })
const users = docs.child('users')

const id = await users.insert({ name: 'Ada', age: 36 })      // generates a time-sortable id
await users.set('123', { name: 'Ada', age: 36 })
await users.update('123', { $set: { name: 'Ada L.' }, $inc: { logins: 1 } })
await users.child('123').update({ $unset: { nickname: true } })   // self form
await users.get('123')
await users.delete('123')
for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) { }
await users.findOne({ email: 'ada@x.io' })
await users.history('123')        // event-sourced only
await users.getAt('123', 17)      // state as of version 17
users.child('123').child('posts') // nested docs, same as KV
```

### 5.1 Events and projections

- The doc at path `P` has its event stream at `P` in `EventStorage`. Its projection is at `~p/P` in the projections KV: `{ state, version }` (and `deleted: true` for tombstones).
- Event types: `{ op: 'set', doc }`, `{ op: 'patch', set?: Record<path, value>, unset?: path[] }`, `{ op: 'delete' }`.
- **Store resolved deltas, not operators.** For example, `$inc` resolves to `{ set: { logins: 6 } }`, so replay is deterministic.
- Supported update operators in v1: `$set`, `$unset`, `$inc`, `$push`, `$pull`, `$addToSet`, with dot-path support.

**Write path:**
1. Read the projection.
2. Compute the new state and patch.
3. `append` with `expectedVersion = projection.version`.
4. Write the projection and index entries (atomic `batch` if available).
5. On `ConcurrencyError`: catch up the projection from the stream, recompute, and retry (bounded).

**Reads** are a single projection `get`. Never replay on read.

**Crash between append and projection write:**
- The projection lags the stream. The next write's `expectedVersion` fails, which triggers catch-up, so the system self-heals.
- Provide `get(key, { consistent: true })` that checks the stream head and catches up first.
- Provide a `catchUp(node?)` maintenance method to repair a subtree.
- `insert` ids are time-sortable (ULID-like). Implement a small generator, with no dependency.

### 5.2 Queries and document indexes

- Filter subset:
  - `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`
  - `$and`, `$or`, `$not`
  - implicit equality and dot paths
- Options: `sort`, `limit`, `skip`, `projection` (fields).
- `find` on a node queries that node's **direct children**.
- Indexes are declared per path pattern (same trie mechanism as 3.3).
  - Each index has `fields` (compound allowed); entries go under `~p/<node>/~i/<name>/<enc(field1)>/<enc(field2)>/…/<childKey>`, using `encodeIndexValue`.
  - `unique: true` additionally claims `~u/<name>/<enc(values)>` via `setIfAbsent`. Throw at construction if the driver lacks `setIfAbsent`.
- **Planner:**
  - Pick the index that covers the most leading equality fields, then at most one range field.
  - Use the index order for `sort` when it matches.
  - Otherwise do a full scan of children projections, with a one-time dev-mode `console.warn` naming the filter.

## 6. Shared conventions

- Every layer uses the same navigation (`child`, `as`, `keys`) and the same self/child arity rules.
- All multi-result reads are `AsyncIterable` (streamed, cancellable). Never build large arrays internally.
- Errors are typed classes exported from `src/errors.ts`: `ConcurrencyError`, `UniqueConstraintError`, `CapabilityError`, `InvalidKeyError`.
- Public exports go through `src/index.ts`, with subpath exports per layer if convenient.

## 7. Types (`src/types/schema.ts`)

Each node is typed by a single **Spec** that describes its own value and its children by key:

```ts
export type Spec = { value?: unknown; children?: Record<string, Spec> }
export type AnySpec = { value: unknown; children: Record<string, AnySpec> }   // untyped default

export type ValueOf<S>    = S extends { value: infer V } ? V : never
export type ChildrenOf<S> = S extends { children: infer C } ? C : {}
export type KeyOf<S>      = Extract<keyof ChildrenOf<S>, string | number>
export type ChildOf<S, K extends KeyOf<S>> = ChildrenOf<S>[K]

// Helpers
export type Many<V, C extends Record<string, Spec> = {}> =
  { children: { [id: string]: { value: V; children: C } } }
export type Fields<T> = { [K in keyof T & string]: { value: T[K] } }
```

Node signatures:

```ts
class KVNode<S extends Spec = AnySpec> {
  child<K extends KeyOf<S>>(key: K): KVNode<ChildOf<S, K>>
  get(): Promise<ValueOf<S> | undefined>
  get<K extends KeyOf<S>>(key: K): Promise<ValueOf<ChildOf<S, K>> | undefined>
  // set/delete/getMany/etc. follow the same pattern
}
```

Example to cover with `expectTypeOf` tests:

```ts
type User = { username: string; age: number }
type Post = { title: string; createdAt: string }
type Schema = { children: {
  users:   Many<User, { posts: Many<Post> }>
  profile: { children: { [id: string]: { children: Fields<User> } } }   // field-per-key style
} }

const db = new KVStorage<Schema>(driver)
db.child('users').get('123')                   // User | undefined
db.child('users').child('123').get()           // User | undefined (same key)
db.child('users').child('123').child('posts')  // KVNode<Many<Post>>
db.child('users').child('123').child('postz')  // ✗ type error
db.child('profile').child('9').get('age')      // number | undefined
db.child('profile').child('9').get('agee')     // ✗ type error
```

Notes:
- **Named levels** check keys at compile time; **dynamic `[id: string]` levels** accept any string.
- One node's children can't mix an index signature with named keys (a TS restriction), so schemas alternate between dynamic and named levels. That alternation exists only in types; the runtime API is just `child()`.
- `EventStorage` and `DocumentStorage` nodes use the same Spec machinery (the value type is the event type or the document type respectively).

## 8. Performance requirements

- Single-key ops: exactly one driver call (no hidden reads), except indexed writes (one `getMany` before the write) and document writes (projection read plus append plus projection write).
- `*Many` ops: one driver call each (plus index work), never N calls.
- `keys()`: skip-scan, O(direct children) seeks.
- Event head lookup: cached; cold lookup is a single reverse scan with `limit: 1`.
- Document `get`: one driver `get`.
- Add `vitest bench` benchmarks for:
  - KV `get`/`set`/`getMany` at 10k/100k keys
  - `keys()` on a node with 1k children each having 100 descendants
  - event append and read
  - document `get`/`update`/indexed `find` vs full-scan `find`
- Record baseline numbers in `DESIGN.md`.

## 9. Milestones

1. **Core:** `path.ts` (escape, unescape, encode, `succ`), `encodeIndexValue`, errors. Property tests for round-trips and order preservation.
2. **KV driver:** `KVDriver`, `BaseKVDriver`, `InMemoryKVDriver`, conformance suite.
3. **KV node:** full API from 3.2, including arity rules, skip-scan `keys`, `deleteTree`, `tree`. Add a model-based property test: random ops vs a reference `Map`.
4. **KV indexes:** pattern trie, atomic and non-atomic write paths, read-repair, range `findBy`.
5. **Events:** `KVEventStorage` with commit keys, `setIfAbsent` and mutex modes, head cache, binary-search `read({ from })`, concurrency tests (simulated interleaving with two storage instances on one driver).
6. **Documents:** `EventSourcedDocumentStorage`, update operators, projections, catch-up, queries, planner, unique indexes, `history` and `getAt`.
7. **Types:** Spec machinery wired through all three layers, plus `expectTypeOf` tests from section 7.
8. **HTTP driver:**
   - A `createKVHttpHandler(driver)` (framework-agnostic `Request → Response`) plus an `HttpKVDriver` client.
   - Endpoints: `POST /getMany`, `POST /setMany`, `POST /deleteMany`, `POST /batch`, `POST /setIfAbsent`, `GET /scan?gte&lt&limit&reverse&values` (NDJSON stream).
   - It must pass the conformance suite end-to-end.
9. **Benchmarks** and a `DESIGN.md` update with results.
10. **(Later, ask first)** `MongoDocumentStorage`:
    - Path pattern maps to a collection (e.g. `users/*/posts` → `users.posts`), with a `_parent` field and a compound index `(_parent, _id)`.
    - Native indexes map from the same index declarations.
    - It should pass the document test suite where semantics allow (no `history`/`getAt`).

## 10. Decisions already made (don't revisit without asking)

- `child()`-only navigation; no collection/doc classes.
- `get()` with no argument reads the node's **own key**. It never assembles the subtree; `tree()` does that explicitly.
- A node's own value and its children are independent keys.
- Self/child dispatch by arity; a passed-but-undefined key throws.
- Indexes are declared on path patterns at construction.
- Events use one key per commit, densely numbered.
- Documents store resolved deltas, and reads hit projections only.
- Keys sort lexicographically.
- Values are JSON-serialisable; byte drivers use a codec.

## 11. Open questions to raise with the owner (don't block on them)

- Order-preserving numeric **keys** (not just index values)?
- Should multi-writer safety be **required** (a hard error without `setIfAbsent`) rather than a documented single-writer fallback?
- The hard-delete story for immutable event logs (compaction or crypto-shredding) for privacy deletion requests.

## 12. Implementation decisions

Decisions taken where the brief was silent. Each is the simplest option consistent with sections 1–10.

### M1 — Core

- **Separator** must be a single printable ASCII character other than `%` and `~`.
- **Empty keys are rejected** (`InvalidKeyError`), as are `NaN`/`±Infinity` number keys. An empty segment
  would produce `a//b`, which is ambiguous with the subtree layout.
- **Separator escaping** uses `%` + two uppercase hex digits (`/` → `%2F`, `%` → `%25`, leading `~` → `%7E`).
- **`InvalidKeyError` extends `TypeError`**, so "passed-but-undefined key throws a `TypeError`" (3.2) and
  the typed error class (6) are the same throw.
- **Index string encoding deviates from "escaped like segments".** Percent-escaping is not
  order-preserving (`"a/"` → `"a%2F"` sorts below `"a&"`, and `"a"` + sep sorts above `"a!"`), which
  would break `findBy` range scans. Instead every character the index encoder emits sorts strictly
  above the separator: code units `<= succ(sep)` are written as `succ(sep)` followed by
  `chr(succ(sep) + 1 + c)`. This is monotone and prefix-safe, so `a < b ⇔ enc(a)+sep+… < enc(b)+sep+…`
  exactly (property-tested). Encodings are still separator-free.
- **Type tags and number digits**: tags are the first five digits of the digit alphabet; numbers use
  16 digits of `0-9a-f` when the separator sorts below `'0'` (e.g. `/`), otherwise the 16 characters
  just above the separator. `-0` is stored as `0`; `NaN` cannot be indexed.
- **Ordering** of strings is by UTF-16 code unit (JS `<`). Drivers that compare UTF-8 bytes agree
  except for astral-plane characters vs. U+E000–U+FFFF; noted as a driver-author caveat.

<!-- decisions appended per milestone -->

## 13. Benchmark baselines

_Filled in at milestone 9._