PotentialStorage/DESIGN.md

PotentialStorage — Design

This document is the design brief agreed with the project owner (sections 0–11), followed by implementation decisions made where the brief was silent (section 12) and benchmark baselines (section 13).

0. Setup and workflow

• Repository: git@git.harvmaster.com:Harvmaster/PotentialStorage.git (SSH; self-hosted Git, not GitHub). Clone it. If you can't reach it, stop and tell the owner rather than starting a fresh repo elsewhere.
• Inspect what's already in the repo first. If it has tooling (package.json, tsconfig, test runner), keep it. If it's empty, set up:
  • TypeScript (strict: true, noUncheckedIndexedAccess: true), ESM, Node ≥ 20
  • Vitest for tests and vitest bench for benchmarks; expectTypeOf for type tests
  • Package name potential-storage unless one already exists
  • No runtime dependencies in the core. Dev dependencies are fine (e.g. fast-check for property tests).
• Copy this brief into the repo as DESIGN.md and create a short CLAUDE.md pointing at it plus the build/test commands.
• Work milestone by milestone (section 9). Commit after each milestone with a clear message, and keep tests green. Ask the owner before pushing.

1. Principles

1. Three layers, each usable on its own: KV → Event (append-only, built on KV) → Document (Mongo-like, built on Events or native).
2. Drivers vs stores. Backends implement a tiny, flat, string-keyed driver. All path, child, index, escaping and typing logic lives once in the user-facing node classes, so every backend gets identical behaviour.
3. One navigation primitive: child(key). There are no collection/document classes. Every node is a store over its direct children, and child(key) re-roots to that key. Navigation is synchronous and does no I/O.
4. No methods on data. Reads return plain values.
5. Performance is a requirement. Batch operations are the primitive; scans are ordered and streamed; nothing replays or scans more than it must (section 8).

2. Paths and key encoding (src/core/path.ts)

• A node's path is readonly string[] of unescaped segments. Number keys are converted with String().
• The separator is a single character, configurable, default /.
• Escaping (per segment): percent-encode %, the separator, and a leading ~. Escaped segments never contain a raw separator. Write escapeSegment / unescapeSegment with round-trip property tests.
• Reserved segments start with ~ (user segments can't, because a leading ~ is escaped):
  • ~i index entries
  • ~u unique-index claims
  • ~e event commits
  • ~p document projections root
• Each node precomputes in its constructor:
  • selfKey = escaped segments joined by the separator (e.g. users/123)
  • prefix = selfKey + sep ('' for the root)
  • No string work per call beyond prefix + escapeSegment(key).
• Keys are ordered lexicographically (so 10 sorts before 9). Document this. An order-preserving numeric key encoding is a possible later option, not part of v1.

Example layout:

users/123                          → KV value of users/123
users/123/posts/p1                 → KV value of a nested child
users/~i/byEmail/sada@x.io/123     → index entry (value: empty / true)
orders/o1/~e/000000000007          → event commit #7 of stream orders/o1
~p/users/123                       → document projection { state, version, commit }

3. Layer 1: KV

3.1 Driver interface (src/kv/driver.ts)

export interface ScanOptions {
  gte?: string
  lt?: string
  limit?: number
  reverse?: boolean
  values?: boolean          // when false/omitted, yield value as undefined
}

export interface KVDriver {
  getMany(keys: string[]): Promise<(unknown | undefined)[]>       // same order as keys
  setMany(entries: [key: string, value: unknown][]): Promise<void>
  deleteMany(keys: string[]): Promise<void>
  scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]>  // MUST be key-ordered

  // Optional capabilities. Stores feature-detect them and fall back where possible.
  setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]>  // true = written
  batch?(ops: BatchOp[]): Promise<void>                            // atomic multi-op write
  deleteRange?(gte: string, lt: string): Promise<void>
}

export type BatchOp =
  | { type: 'set'; key: string; value: unknown }
  | { type: 'delete'; key: string }

• Values are passed through as JSON-serialisable values. Drivers that need bytes (HTTP, Redis, files) serialise internally via a pluggable Codec (default JSON).
• Provide an abstract BaseKVDriver with default implementations of deleteRange (scan + deleteMany) and helpers, so a minimal driver implements only the four required methods.
• InMemoryKVDriver: a sorted structure (sorted array with binary search is fine for v1; keep it behind an interface so it can become a B-tree). It implements every optional capability: setIfAbsent, atomic batch and deleteRange.
• Write a reusable driver conformance suite runKVDriverConformance(name, makeDriver) covering ordering, gte/lt bounds, reverse, limit, values on/off, getMany order with missing keys, and each optional capability when present.

3.2 KVStorage / KVNode (src/kv/node.ts)

const db = new KVStorage(driver, { separator: '/', indexes: { ... } })   // db is the root KVNode
const users = db.child('users')
const user  = users.child('123')

API of every KVNode (types in section 7):

Method	Meaning
child(key)	Sync. Returns a node at path + key.
as<S>()	Type assertion for untyped use; returns the same node retyped.
get() / get(key)	Own value at selfKey / value at child key.
getMany(keys)	Batch child read, same order, undefined for missing.
set(value) / set(key, value)	Write own value / child value.
setMany(entries)	Batch child write.
delete() / delete(key)	Delete own value / child value only (not descendants).
deleteMany(keys)	Batch child delete.
deleteTree() / deleteTree(key)	Delete own value and all descendants (uses deleteRange if available).
keys(opts?)	AsyncIterable<string> of direct child keys (unescaped). Options: limit, after, reverse.
entries(opts?)	AsyncIterable<[key, value]> of direct children that have an own value.
tree({ depth })	Assemble an object from descendant keys up to depth (for field-per-key style). Bounded and explicit; never the default.
findBy(index, value | { gte?, lt? }, opts?)	Index lookup over direct children (section 3.3).

Arity dispatch (important):

• Self vs child is decided by arguments.length, never by checking for undefined.
• If a key argument is passed but is undefined or null, throw a TypeError (this prevents users.get(maybeId) silently reading self).
• The root node (path = []) has no own value: get(), set(v) and delete() on root throw.
• Implement one private #resolve(args) that every method uses.

Direct-children listing (keys) uses a skip-scan:

1. Scan gte: prefix, lt: prefix + succ(sep).
2. For each key, take the first segment after prefix. Skip reserved (~…) segments. Yield the segment (unescaped) if it differs from the previous one.
3. When the next key is a descendant of the segment just yielded, re-seek the driver to gte: prefix + escSeg + succ(sep) (where succ(c) is the next char code) instead of iterating through the subtree. Keep using the same iterator while keys are siblings.

Listing must be O(direct children × seek cost), not O(subtree). A child that has no own value but has descendants is listed.

3.3 Indexes (src/kv/indexes.ts)

Indexes are declared on path patterns in the KVStorage constructor. A pattern names a parent node, and its index applies to that node's direct children's values:

new KVStorage(driver, {
  indexes: {
    'users':         { byEmail: (u: User) => u.email },
    'users/*/posts': { byDate:  (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
  },
})

• An index function returns IndexValue | IndexValue[] | undefined, where IndexValue = string | number | boolean | null. An array means multi-entry; undefined means not indexed.
• Compile patterns into a segment trie (* matches one segment). A node resolves its index set once, at construction, in O(depth).
• Entry key: <nodePrefix>~i/<indexName>/<encodedValue>/<escapedChildKey> with a true value.
• encodeIndexValue must be order-preserving and separator-safe:
  • Type tag first, so values order by type: null < false < true < numbers < strings.
  • Numbers use an order-preserving float64 encoding (flip the sign bit for positives and all bits for negatives, then fixed-width hex).
  • Strings are escaped like segments.
  • This enables findBy(index, { gte, lt }) range scans.
• Writes to an indexed node:
  1. Read the old value(s) (getMany) and compute the diff of index entries.
  2. If the driver has batch, write the value and the index adds/removes atomically.
  3. Otherwise, write index adds first, then the value, then index removes. Treat index entries as hints: findBy re-reads each value, re-checks the index function, and lazily deletes stale entries (read-repair).
• Non-indexed nodes must not pay the read-before-write cost.

4. Layer 2: Events (src/events/)

An append-only log per stream. A stream is identified by a node path. Same child() navigation as KV.

const events = new EventStorage(kv)            // takes a KVStorage (or a KVNode as root)
const orders = events.child('orders')

await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 })
await orders.child('o1').append([{ type: 'paid' }])          // self form
for await (const e of orders.read('o1', { from: 1 })) { /* { version, commit, at, data } */ }
await orders.version('o1')
for await (const id of orders.keys()) { }                    // child streams
orders.child('o1').child('shipments')

• EventStorage is the abstract interface. KVEventStorage is the KV-backed implementation. Leave room for native backends (EventStoreDB, Kafka) implementing the same interface.
• Arity: append(events, opts?) vs append(key, events, opts?). Dispatch on Array.isArray(args[0]), plus the undefined-key rule from 3.2. read, version and keys follow the 3.2 rules.
• Versions: an empty stream has version 0, and the first event has version 1. expectedVersion: number | 'any' (default 'any').
• Storage: one key per commit, densely numbered.
  • Key: <streamSelfKey>/~e/<commitSeq zero-padded to 12>
  • Value: { firstVersion, at, events: E[] }
  • The commit sequence (not the event version) is the key so that any stale writer collides with the existing key. Keying by version would let a stale writer create overlapping version ranges.
• Append:
  1. Get the head { commitSeq, version } (from the cache, or a reverse scan with limit: 1).
  2. If expectedVersion is a number and doesn't match the head, throw ConcurrencyError.
  3. Call setIfAbsent(nextCommitKey).
  4. On false (collision): refresh the head. If expectedVersion was given, throw ConcurrencyError; otherwise retry up to 5 times.
  5. Without setIfAbsent, fall back to a per-stream in-process mutex. Document that this mode is single-writer only.
• Head cache: a bounded LRU keyed by stream path. Update it on successful append; invalidate it on collision.
• read({ from, to, reverse, limit }) streams events with metadata. To start at version from, locate the containing commit by binary search over commit numbers using getMany probes (O(log commits) round trips), then forward-scan.
• There is no update or delete. Keys inside ~e are never rewritten.

5. Layer 3: Documents (src/documents/)

DocumentStorage is the interface. There are two implementations:

• EventSourcedDocumentStorage(events, { projections?: KVNode }) is the v1 target. Projections default to the ~p root on the same KV.
• MongoDocumentStorage(client) is a later milestone (section 9).

const docs  = new EventSourcedDocumentStorage(events, { indexes: { 'users': [{ fields: ['email'], unique: true }, { fields: ['age'] }] } })
const users = docs.child('users')

const id = await users.insert({ name: 'Ada', age: 36 })      // generates a time-sortable id
await users.set('123', { name: 'Ada', age: 36 })
await users.update('123', { $set: { name: 'Ada L.' }, $inc: { logins: 1 } })
await users.child('123').update({ $unset: { nickname: true } })   // self form
await users.get('123')
await users.delete('123')
for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) { }
await users.findOne({ email: 'ada@x.io' })
await users.history('123')        // event-sourced only
await users.getAt('123', 17)      // state as of version 17
users.child('123').child('posts') // nested docs, same as KV

5.1 Events and projections

• The doc at path P has its event stream at P in EventStorage. Its projection is at ~p/P in the projections KV: { state, version } (and deleted: true for tombstones).
• Event types: { op: 'set', doc }, { op: 'patch', set?: Record<path, value>, unset?: path[] }, { op: 'delete' }.
• Store resolved deltas, not operators. For example, $inc resolves to { set: { logins: 6 } }, so replay is deterministic.
• Supported update operators in v1: $set, $unset, $inc, $push, $pull, $addToSet, with dot-path support.

Write path:

1. Read the projection.
2. Compute the new state and patch.
3. append with expectedVersion = projection.version.
4. Write the projection and index entries (atomic batch if available).
5. On ConcurrencyError: catch up the projection from the stream, recompute, and retry (bounded).

Reads are a single projection get. Never replay on read.

Crash between append and projection write:

• The projection lags the stream. The next write's expectedVersion fails, which triggers catch-up, so the system self-heals.
• Provide get(key, { consistent: true }) that checks the stream head and catches up first.
• Provide a catchUp(node?) maintenance method to repair a subtree.
• insert ids are time-sortable (ULID-like). Implement a small generator, with no dependency.

5.2 Queries and document indexes

• Filter subset:
  • $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $exists
  • $and, $or, $not
  • implicit equality and dot paths
• Options: sort, limit, skip, projection (fields).
• find on a node queries that node's direct children.
• Indexes are declared per path pattern (same trie mechanism as 3.3).
  • Each index has fields (compound allowed); entries go under ~p/<node>/~i/<name>/<enc(field1)>/<enc(field2)>/…/<childKey>, using encodeIndexValue.
  • unique: true additionally claims ~u/<name>/<enc(values)> via setIfAbsent. Throw at construction if the driver lacks setIfAbsent.
• Planner:
  • Pick the index that covers the most leading equality fields, then at most one range field.
  • Use the index order for sort when it matches.
  • Otherwise do a full scan of children projections, with a one-time dev-mode console.warn naming the filter.

6. Shared conventions

• Every layer uses the same navigation (child, as, keys) and the same self/child arity rules.
• All multi-result reads are AsyncIterable (streamed, cancellable). Never build large arrays internally.
• Errors are typed classes exported from src/errors.ts: ConcurrencyError, UniqueConstraintError, CapabilityError, InvalidKeyError.
• Public exports go through src/index.ts, with subpath exports per layer if convenient.

7. Types (src/types/schema.ts)

Each node is typed by a single Spec that describes its own value and its children by key:

export type Spec = { value?: unknown; children?: Record<string, Spec> }
export type AnySpec = { value: unknown; children: Record<string, AnySpec> }   // untyped default

export type ValueOf<S>    = S extends { value: infer V } ? V : never
export type ChildrenOf<S> = S extends { children: infer C } ? C : {}
export type KeyOf<S>      = Extract<keyof ChildrenOf<S>, string | number>
export type ChildOf<S, K extends KeyOf<S>> = ChildrenOf<S>[K]

// Helpers
export type Many<V, C extends Record<string, Spec> = {}> =
  { children: { [id: string]: { value: V; children: C } } }
export type Fields<T> = { [K in keyof T & string]: { value: T[K] } }

Node signatures:

class KVNode<S extends Spec = AnySpec> {
  child<K extends KeyOf<S>>(key: K): KVNode<ChildOf<S, K>>
  get(): Promise<ValueOf<S> | undefined>
  get<K extends KeyOf<S>>(key: K): Promise<ValueOf<ChildOf<S, K>> | undefined>
  // set/delete/getMany/etc. follow the same pattern
}

Example to cover with expectTypeOf tests:

type User = { username: string; age: number }
type Post = { title: string; createdAt: string }
type Schema = { children: {
  users:   Many<User, { posts: Many<Post> }>
  profile: { children: { [id: string]: { children: Fields<User> } } }   // field-per-key style
} }

const db = new KVStorage<Schema>(driver)
db.child('users').get('123')                   // User | undefined
db.child('users').child('123').get()           // User | undefined (same key)
db.child('users').child('123').child('posts')  // KVNode<Many<Post>>
db.child('users').child('123').child('postz')  // ✗ type error
db.child('profile').child('9').get('age')      // number | undefined
db.child('profile').child('9').get('agee')     // ✗ type error

Notes:

• Named levels check keys at compile time; dynamic [id: string] levels accept any string.
• One node's children can't mix an index signature with named keys (a TS restriction), so schemas alternate between dynamic and named levels. That alternation exists only in types; the runtime API is just child().
• EventStorage and DocumentStorage nodes use the same Spec machinery (the value type is the event type or the document type respectively).

8. Performance requirements

• Single-key ops: exactly one driver call (no hidden reads), except indexed writes (one getMany before the write) and document writes (projection read plus append plus projection write).
• *Many ops: one driver call each (plus index work), never N calls.
• keys(): skip-scan, O(direct children) seeks.
• Event head lookup: cached; cold lookup is a single reverse scan with limit: 1.
• Document get: one driver get.
• Add vitest bench benchmarks for:
  • KV get/set/getMany at 10k/100k keys
  • keys() on a node with 1k children each having 100 descendants
  • event append and read
  • document get/update/indexed find vs full-scan find
• Record baseline numbers in DESIGN.md.

9. Milestones

1. Core: path.ts (escape, unescape, encode, succ), encodeIndexValue, errors. Property tests for round-trips and order preservation.
2. KV driver: KVDriver, BaseKVDriver, InMemoryKVDriver, conformance suite.
3. KV node: full API from 3.2, including arity rules, skip-scan keys, deleteTree, tree. Add a model-based property test: random ops vs a reference Map.
4. KV indexes: pattern trie, atomic and non-atomic write paths, read-repair, range findBy.
5. Events: KVEventStorage with commit keys, setIfAbsent and mutex modes, head cache, binary-search read({ from }), concurrency tests (simulated interleaving with two storage instances on one driver).
6. Documents: EventSourcedDocumentStorage, update operators, projections, catch-up, queries, planner, unique indexes, history and getAt.
7. Types: Spec machinery wired through all three layers, plus expectTypeOf tests from section 7.
8. HTTP driver:
   • A createKVHttpHandler(driver) (framework-agnostic Request → Response) plus an HttpKVDriver client.
   • Endpoints: POST /getMany, POST /setMany, POST /deleteMany, POST /batch, POST /setIfAbsent, GET /scan?gte&lt&limit&reverse&values (NDJSON stream).
   • It must pass the conformance suite end-to-end.
9. Benchmarks and a DESIGN.md update with results.
10. (Later, ask first) MongoDocumentStorage:
    • Path pattern maps to a collection (e.g. users/*/posts → users.posts), with a _parent field and a compound index (_parent, _id).
    • Native indexes map from the same index declarations.
    • It should pass the document test suite where semantics allow (no history/getAt).

10. Decisions already made (don't revisit without asking)

• child()-only navigation; no collection/doc classes.
• get() with no argument reads the node's own key. It never assembles the subtree; tree() does that explicitly.
• A node's own value and its children are independent keys.
• Self/child dispatch by arity; a passed-but-undefined key throws.
• Indexes are declared on path patterns at construction.
• Events use one key per commit, densely numbered.
• Documents store resolved deltas, and reads hit projections only.
• Keys sort lexicographically.
• Values are JSON-serialisable; byte drivers use a codec.

11. Open questions to raise with the owner (don't block on them)

• Order-preserving numeric keys (not just index values)?
• Should multi-writer safety be required (a hard error without setIfAbsent) rather than a documented single-writer fallback?
• The hard-delete story for immutable event logs (compaction or crypto-shredding) for privacy deletion requests.

12. Implementation decisions

Decisions taken where the brief was silent. Each is the simplest option consistent with sections 1–10.

M1 — Core

• Separator must be a single printable ASCII character other than % and ~.
• Empty keys are rejected (InvalidKeyError), as are NaN/±Infinity number keys. An empty segment would produce a//b, which is ambiguous with the subtree layout.
• Separator escaping uses % + two uppercase hex digits (/ → %2F, % → %25, leading ~ → %7E).
• InvalidKeyError extends TypeError, so "passed-but-undefined key throws a TypeError" (3.2) and the typed error class (6) are the same throw.
• Index string encoding deviates from "escaped like segments". Percent-escaping is not order-preserving ("a/" → "a%2F" sorts below "a&", and "a" + sep sorts above "a!"), which would break findBy range scans. Instead every character the index encoder emits sorts strictly above the separator: code units <= succ(sep) are written as succ(sep) followed by chr(succ(sep) + 1 + c). This is monotone and prefix-safe, so a < b ⇔ enc(a)+sep+… < enc(b)+sep+… exactly (property-tested). Encodings are still separator-free.
• Type tags and number digits: tags are the first five digits of the digit alphabet; numbers use 16 digits of 0-9a-f when the separator sorts below '0' (e.g. /), otherwise the 16 characters just above the separator. -0 is stored as 0; NaN cannot be indexed.
• Ordering of strings is by UTF-16 code unit (JS <). Drivers that compare UTF-8 bytes agree except for astral-plane characters vs. U+E000–U+FFFF; noted as a driver-author caveat.

M2 — KV driver

• Conformance suite location: src/testing/kv-conformance.ts, exported as potential-storage/testing. It imports vitest, which is an optional peer dependency; the core entry point never imports it.
• InMemoryKVDriver clones values on write and read (structuredClone, skipped for primitives) so stored state can't be mutated through shared references, matching serialising drivers. Opt out with { clone: false }.
• Scans tolerate concurrent writes: the in-memory scan remembers the last key yielded and re-seeks if the map changed while suspended. Read-repair and deleteTree rely on this. The conformance suite requires it of every driver (deleting an already-yielded key mid-scan must not skip keys).
• setMany bulk path: ≥32 new keys are sorted and merged (O(n + k log k)); fewer are spliced.
• BaseKVDriver.deleteRange deletes in chunks of 1000 (scan with limit, then deleteMany, repeat), so it never deletes from under its own live iterator. It also provides get/set/delete single-key helpers.
• No fake batch: BaseKVDriver deliberately does not provide a non-atomic batch, since stores use its presence to mean "atomic".

M3 — KV node

• Child order is "subtree order", not plain segment order. The brief's skip-scan assumes a child's keys are contiguous, but they aren't when a sibling extends a segment with a character that sorts below the separator (for /: space and !"#$%&'()*+,-.). With children a and a-1 the key space is p/a, p/a-1, p/a-1/x, p/a/x, so a's descendants sort after a-1, and "yield if different from the previous segment" would emit a twice. keys()/entries() therefore order children by escaped segment followed by the separator (a-1/ < a/), which is the order of their subtrees in key space. For keys without such characters this is plain lexicographic order.
  • Forward listing defers a child seen only through its own key on a small stack (bounded by segment length) until the scan passes p/<seg>/. It costs no extra driver calls beyond the brief's one seek per child with descendants.
  • Reverse listing needs one batched getMany probe per new child whose segment contains a below-separator character, to find shorter siblings that only have an own key.
  • after is exclusive in the direction of travel and follows the same order.
  • Implemented once in src/kv/children.ts (scanChildren) and reused by the event and document layers. A model-based property test checks forward, reverse, after and limit against brute force with keys like a, a-1, a.b, a/x, %, ~t.
  • Open question for the owner: keep subtree order, or change segment escaping so the key layout nests exactly. That would mean escaping every below-separator character with an escape character above the separator, at the cost of the readable percent-encoding.
• Errors from async methods are rejections. Every async method (get, set, …) is an async function, so a bad key yields a rejected promise rather than a synchronous throw. child() is sync and throws synchronously.
• set(undefined) is rejected (TypeError: use delete()); undefined means "missing" in reads.
• deleteTree() on the root is allowed and clears the whole store (chunked scan + delete, since the root range has no upper bound). Subtree deletion is delete(selfKey) plus deleteRange(prefix, selfKey + succ(sep)). It is not [selfKey, …), which would also catch siblings such as a!.
• tree({ depth }) builds a plain object: leaves are values, interior nodes are objects. A node with both an own value and children exposes its own value under the '' property (empty keys are invalid, so it can't collide). Reserved subtrees are skipped. Deeper levels are skipped by seeking rather than reading.
• keys() and entries() yield string keys (number keys come back as strings).
• Exposed low-level properties on every node: path, selfKey, prefix, end (subtree upper bound), driver, separator, isRoot. The event and document layers build on these.

13. Benchmark baselines

Filled in at milestone 9.