Harvmaster/PotentialStorage
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: Harvmaster/PotentialStorage:main
Branches Tags
Harvmaster/PotentialStorage:main Harvmaster/PotentialStorage:sync Harvmaster/PotentialStorage:simplify
...
pull from: Harvmaster/PotentialStorage:simplify
Branches Tags
Harvmaster/PotentialStorage:sync Harvmaster/PotentialStorage:simplify Harvmaster/PotentialStorage:main
1 Commits
main ... simplify
Author SHA1 Message Date
Claude 08fb78721f Simplify: one abstract base class per layer
CI / Node 20 (pull_request) Successful in 10s
Details
CI / Node 22 (pull_request) Successful in 11s
Details 
- BaseKVStorage / BaseEventStorage / BaseDocumentStorage define the API once;
  implementations provide only raw primitives (client and server share an API)
- Drop KV indexes, query planner, unique indexes; document stores take index fields
- Escape every char <= '/' so child keys stay contiguous; skip-scan is now simple
- Event-sourced documents = event storage + any document store for projections
- New folder layout (kv-storage, event-storage, document-storage, utils); tests only in test/

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
 2026-10-23 08:05:03 +00:00
80 changed files with 2518 additions and 6337 deletions
Expand all files Collapse all files
CLAUDE.md
+16 -16
View File
@@ -1,24 +1,24 @@
# PotentialStorage
Layered TypeScript storage library: KV → Events → Documents. The full design, milestones and
implementation decisions are in [DESIGN.md](./DESIGN.md) — read it before changing anything.
Record new decisions in DESIGN.md section 12.
Layered TypeScript storage library: KV → Events → Documents. The design is in
[DESIGN.md](./DESIGN.md); read it before changing anything and keep it up to date.
## Commands
- `npm test` — `tsc --noEmit` then all Vitest tests (including `*.test-d.ts` type tests)
- `npm run test:only -- <file>` — Vitest only, e.g. a single file
- `npm run typecheck` — `tsc --noEmit` over src, test and bench
- `npm run bench` — `vitest bench`
- `npm run build` — emit `dist/`
- `npm run format` / `npm run format:check` — Prettier
- `npm test`: `tsc --noEmit`, then all Vitest tests (including `test/types.test-d.ts`)
- `npm run test:only -- <file>`: Vitest only
- `npm run typecheck`, `npm run build`, `npm run bench`
- `npm run format` / `npm run format:check`: Prettier
CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs format check, typecheck, tests and build on Node 20 and 22.
CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs the format check, typecheck, tests and build on Node 20 and 22.
## Rules
## Conventions
- No runtime dependencies in `src/`. Dev dependencies only.
- Drivers stay flat and string-keyed; all path/index/escaping logic lives in the node classes.
- Self vs child dispatch is by `arguments.length`; a passed-but-`undefined`/`null` key throws.
- Multi-result reads are `AsyncIterable`; don't build large arrays internally.
- Keep tests green and commit per milestone. Ask the owner before pushing.
- One abstract base class per layer (`src/<layer>/base-*.ts`); implementations only provide the raw
primitives, and the user-facing API lives in the base class.
- Classes over loose functions; TS `private`, not `#`; no overloads (use `param?` or tuple-union
rest params); don't override base methods in subclasses.
- No runtime dependencies. Tests live only in `test/`, mirroring `src/`.
- New KV/document implementations must pass `test/kv-storage/kv-storage.conformance.ts` /
`test/document-storage/document-storage.suite.ts`.
- Ask the owner before pushing.
DESIGN.md
+165 -644
View File
@@ -1,655 +1,176 @@
# PotentialStorage — Design
This document is the design brief agreed with the project owner (sections 0–11), followed by
implementation decisions made where the brief was silent (section 12) and benchmark baselines
(section 13).
Layered TypeScript storage: **KV → Events → Documents**. Each layer is an abstract base class
that implementations extend; each is usable on its own. No runtime dependencies.
## 0. Setup and workflow
- Repository: `git@git.harvmaster.com:Harvmaster/PotentialStorage.git` (SSH; self-hosted Git, not GitHub). Clone it. If you can't reach it, stop and tell the owner rather than starting a fresh repo elsewhere.
- Inspect what's already in the repo first. If it has tooling (package.json, tsconfig, test runner), keep it. If it's empty, set up:
- TypeScript (`strict: true`, `noUncheckedIndexedAccess: true`), ESM, Node ≥ 20
- Vitest for tests and `vitest bench` for benchmarks; `expectTypeOf` for type tests
- Package name `potential-storage` unless one already exists
- **No runtime dependencies in the core.** Dev dependencies are fine (e.g. `fast-check` for property tests).
- Copy this brief into the repo as `DESIGN.md` and create a short `CLAUDE.md` pointing at it plus the build/test commands.
- Work milestone by milestone (section 9). Commit after each milestone with a clear message, and keep tests green. Ask the owner before pushing.
This supersedes the original implementation brief (see git history before the `simplify` branch);
section 7 lists what changed and why.
## 1. Principles
1. **Three layers**, each usable on its own: **KV** → **Event** (append-only, built on KV) → **Document** (Mongo-like, built on Events or native).
2. **Drivers vs stores.** Backends implement a tiny, flat, string-keyed _driver_. All path, child, index, escaping and typing logic lives once in the user-facing _node_ classes, so every backend gets identical behaviour.
3. **One navigation primitive: `child(key)`.** There are no collection/document classes. Every node is a store over its direct children, and `child(key)` re-roots to that key. Navigation is synchronous and does no I/O.
4. **No methods on data.** Reads return plain values.
5. **Performance is a requirement.** Batch operations are the primitive; scans are ordered and streamed; nothing replays or scans more than it must (section 8).
1. **One abstract class per layer.** Implementations provide a handful of raw primitives; the
whole user-facing API is written once in the base class. A client (e.g. HTTP) and a server-side
store are the same kind of object, and wrappers (e.g. encryption) are just another subclass
that delegates the primitives.
2. **One navigation primitive: `child(key)`.** Every instance is a node; `child` returns a view of
the same storage rooted deeper (via `Object.create`, so it shares the instance's state). Sync, no I/O.
3. **Self or child by argument.** `get()` reads this node, `get(key)` a child. A key that is passed
but `undefined`/`null` throws (`InvalidKeyError`) instead of silently meaning "self". Methods use
optional parameters or tuple-union rest parameters, not overloads.
4. **Indexes belong to the store.** Document stores receive the fields to index and use them however
suits the backend (in-memory equality maps, native Mongo/SQLite/IndexedDB indexes, or nothing).
5. **Simple over clever.** Code style: classes, TS `private` (not `#`), no method overrides in
subclasses, tests only under `test/`.
## 2. Paths and key encoding (`src/core/path.ts`)
- A node's path is `readonly string[]` of **unescaped** segments. Number keys are converted with `String()`.
- The separator is a single character, configurable, default `/`.
- **Escaping** (per segment): percent-encode `%`, the separator, and a **leading** `~`. Escaped segments never contain a raw separator. Write `escapeSegment` / `unescapeSegment` with round-trip property tests.
- **Reserved segments** start with `~` (user segments can't, because a leading `~` is escaped):
- `~i` index entries
- `~u` unique-index claims
- `~e` event commits
- `~p` document projections root
- Each node precomputes in its constructor:
- `selfKey` = escaped segments joined by the separator (e.g. `users/123`)
- `prefix` = `selfKey + sep` (`''` for the root)
- No string work per call beyond `prefix + escapeSegment(key)`.
- Keys are ordered **lexicographically** (so `10` sorts before `9`). Document this. An order-preserving numeric key encoding is a possible later option, not part of v1.
Example layout:
## 2. Layout
```
users/123 → KV value of users/123
users/123/posts/p1 → KV value of a nested child
users/~i/byEmail/sada@x.io/123 → index entry (value: empty / true)
orders/o1/~e/000000000007 → event commit #7 of stream orders/o1
~p/users/123 → document projection { state, version, commit }
src/
kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-http.ts, kv-storage-http-server.ts
event-storage/ base-event-storage.ts, event-storage-kv.ts
document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts,
event-sourced-document-storage.ts
utils/ keys, sorted-map, json, ulid, errors, schema (types), document-path/query/update
test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts
bench/
```
## 3. Layer 1: KV
### 3.1 Driver interface (`src/kv/driver.ts`)
```ts
export interface ScanOptions {
gte?: string
lt?: string
limit?: number
reverse?: boolean
values?: boolean // when false/omitted, yield value as undefined
}
export interface KVDriver {
getMany(keys: string[]): Promise<(unknown | undefined)[]> // same order as keys
setMany(entries: [key: string, value: unknown][]): Promise<void>
deleteMany(keys: string[]): Promise<void>
scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]> // MUST be key-ordered
// Optional capabilities. Stores feature-detect them and fall back where possible.
setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]> // true = written
batch?(ops: BatchOp[]): Promise<void> // atomic multi-op write
deleteRange?(gte: string, lt: string): Promise<void>
}
export type BatchOp = { type: 'set'; key: string; value: unknown } | { type: 'delete'; key: string }
```
- Values are passed through as JSON-serialisable values. Drivers that need bytes (HTTP, Redis, files) serialise internally via a pluggable `Codec` (default JSON).
- Provide an abstract `BaseKVDriver` with default implementations of `deleteRange` (scan + deleteMany) and helpers, so a minimal driver implements only the four required methods.
- **`InMemoryKVDriver`**: a sorted structure (sorted array with binary search is fine for v1; keep it behind an interface so it can become a B-tree). It implements every optional capability: `setIfAbsent`, atomic `batch` and `deleteRange`.
- Write a reusable **driver conformance suite** `runKVDriverConformance(name, makeDriver)` covering ordering, `gte`/`lt` bounds, reverse, limit, `values` on/off, `getMany` order with missing keys, and each optional capability when present.
### 3.2 `KVStorage` / `KVNode` (`src/kv/node.ts`)
```ts
const db = new KVStorage(driver, { separator: '/', indexes: { ... } }) // db is the root KVNode
const users = db.child('users')
const user = users.child('123')
```
API of every `KVNode` (types in section 7):
| Method | Meaning |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `child(key)` | Sync. Returns a node at `path + key`. |
| `as<S>()` | Type assertion for untyped use; returns the same node retyped. |
| `get()` / `get(key)` | Own value at `selfKey` / value at child key. |
| `getMany(keys)` | Batch child read, same order, `undefined` for missing. |
| `set(value)` / `set(key, value)` | Write own value / child value. |
| `setMany(entries)` | Batch child write. |
| `delete()` / `delete(key)` | Delete own value / child value only (not descendants). |
| `deleteMany(keys)` | Batch child delete. |
| `deleteTree()` / `deleteTree(key)` | Delete own value **and all descendants** (uses `deleteRange` if available). |
| `keys(opts?)` | `AsyncIterable<string>` of **direct** child keys (unescaped). Options: `limit`, `after`, `reverse`. |
| `entries(opts?)` | `AsyncIterable<[key, value]>` of direct children that have an own value. |
| `tree({ depth })` | Assemble an object from descendant keys up to `depth` (for field-per-key style). Bounded and explicit; never the default. |
| `findBy(index, value \| { gte?, lt? }, opts?)` | Index lookup over direct children (section 3.3). |
**Arity dispatch (important):**
- Self vs child is decided by `arguments.length`, never by checking for `undefined`.
- If a key argument is passed but is `undefined` or `null`, throw a `TypeError` (this prevents `users.get(maybeId)` silently reading self).
- The root node (`path = []`) has no own value: `get()`, `set(v)` and `delete()` on root throw.
- Implement one private `#resolve(args)` that every method uses.
**Direct-children listing (`keys`)** uses a skip-scan:
1. Scan `gte: prefix`, `lt: prefix + succ(sep)`.
2. For each key, take the first segment after `prefix`. Skip reserved (`~…`) segments. Yield the segment (unescaped) if it differs from the previous one.
3. When the next key is a descendant of the segment just yielded, re-seek the driver to `gte: prefix + escSeg + succ(sep)` (where `succ(c)` is the next char code) instead of iterating through the subtree. Keep using the same iterator while keys are siblings.
Listing must be O(direct children × seek cost), not O(subtree). A child that has no own value but has descendants **is** listed.
### 3.3 Indexes (`src/kv/indexes.ts`)
Indexes are declared on **path patterns** in the `KVStorage` constructor. A pattern names a parent node, and its index applies to that node's direct children's values:
```ts
new KVStorage(driver, {
indexes: {
users: { byEmail: (u: User) => u.email },
'users/*/posts': { byDate: (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
},
})
```
- An index function returns `IndexValue | IndexValue[] | undefined`, where `IndexValue = string | number | boolean | null`. An array means multi-entry; `undefined` means not indexed.
- Compile patterns into a **segment trie** (`*` matches one segment). A node resolves its index set **once, at construction**, in O(depth).
- Entry key: `<nodePrefix>~i/<indexName>/<encodedValue>/<escapedChildKey>` with a `true` value.
- **`encodeIndexValue` must be order-preserving and separator-safe:**
- Type tag first, so values order by type: `null` < `false` < `true` < numbers < strings.
- Numbers use an order-preserving float64 encoding (flip the sign bit for positives and all bits for negatives, then fixed-width hex).
- Strings are escaped like segments.
- This enables `findBy(index, { gte, lt })` range scans.
- **Writes to an indexed node:**
1. Read the old value(s) (`getMany`) and compute the diff of index entries.
2. If the driver has `batch`, write the value and the index adds/removes atomically.
3. Otherwise, write index **adds first**, then the value, then index **removes**. Treat index entries as hints: `findBy` re-reads each value, re-checks the index function, and lazily deletes stale entries (read-repair).
- Non-indexed nodes must **not** pay the read-before-write cost.
## 4. Layer 2: Events (`src/events/`)
An append-only log per stream. A stream is identified by a node path. Same `child()` navigation as KV.
```ts
const events = new EventStorage(kv) // takes a KVStorage (or a KVNode as root)
const orders = events.child('orders')
await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 })
await orders.child('o1').append([{ type: 'paid' }]) // self form
for await (const e of orders.read('o1', { from: 1 })) {
/* { version, commit, at, data } */
}
await orders.version('o1')
for await (const id of orders.keys()) {
} // child streams
orders.child('o1').child('shipments')
```
- `EventStorage` is the abstract interface. `KVEventStorage` is the KV-backed implementation. Leave room for native backends (EventStoreDB, Kafka) implementing the same interface.
- **Arity:** `append(events, opts?)` vs `append(key, events, opts?)`. Dispatch on `Array.isArray(args[0])`, plus the undefined-key rule from 3.2. `read`, `version` and `keys` follow the 3.2 rules.
- **Versions:** an empty stream has version 0, and the first event has version 1. `expectedVersion: number | 'any'` (default `'any'`).
- **Storage: one key per commit, densely numbered.**
- Key: `<streamSelfKey>/~e/<commitSeq zero-padded to 12>`
- Value: `{ firstVersion, at, events: E[] }`
- The commit sequence (not the event version) is the key so that **any** stale writer collides with the existing key. Keying by version would let a stale writer create overlapping version ranges.
- **Append:**
1. Get the head `{ commitSeq, version }` (from the cache, or a reverse scan with `limit: 1`).
2. If `expectedVersion` is a number and doesn't match the head, throw `ConcurrencyError`.
3. Call `setIfAbsent(nextCommitKey)`.
4. On `false` (collision): refresh the head. If `expectedVersion` was given, throw `ConcurrencyError`; otherwise retry up to 5 times.
5. Without `setIfAbsent`, fall back to a per-stream in-process mutex. Document that this mode is single-writer only.
- **Head cache:** a bounded LRU keyed by stream path. Update it on successful append; invalidate it on collision.
- **`read({ from, to, reverse, limit })`** streams events with metadata. To start at version `from`, locate the containing commit by binary search over commit numbers using `getMany` probes (O(log commits) round trips), then forward-scan.
- There is no update or delete. Keys inside `~e` are never rewritten.
## 5. Layer 3: Documents (`src/documents/`)
`DocumentStorage` is the interface. There are two implementations:
- `EventSourcedDocumentStorage(events, { projections?: KVNode })` is the v1 target. Projections default to the `~p` root on the same KV.
- `MongoDocumentStorage(client)` is a later milestone (section 9).
```ts
const docs = new EventSourcedDocumentStorage(events, { indexes: { users: [{ fields: ['email'], unique: true }, { fields: ['age'] }] } })
const users = docs.child('users')
const id = await users.insert({ name: 'Ada', age: 36 }) // generates a time-sortable id
await users.set('123', { name: 'Ada', age: 36 })
await users.update('123', { $set: { name: 'Ada L.' }, $inc: { logins: 1 } })
await users.child('123').update({ $unset: { nickname: true } }) // self form
await users.get('123')
await users.delete('123')
for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) {
}
await users.findOne({ email: 'ada@x.io' })
await users.history('123') // event-sourced only
await users.getAt('123', 17) // state as of version 17
users.child('123').child('posts') // nested docs, same as KV
```
### 5.1 Events and projections
- The doc at path `P` has its event stream at `P` in `EventStorage`. Its projection is at `~p/P` in the projections KV: `{ state, version }` (and `deleted: true` for tombstones).
- Event types: `{ op: 'set', doc }`, `{ op: 'patch', set?: Record<path, value>, unset?: path[] }`, `{ op: 'delete' }`.
- **Store resolved deltas, not operators.** For example, `$inc` resolves to `{ set: { logins: 6 } }`, so replay is deterministic.
- Supported update operators in v1: `$set`, `$unset`, `$inc`, `$push`, `$pull`, `$addToSet`, with dot-path support.
**Write path:**
1. Read the projection.
2. Compute the new state and patch.
3. `append` with `expectedVersion = projection.version`.
4. Write the projection and index entries (atomic `batch` if available).
5. On `ConcurrencyError`: catch up the projection from the stream, recompute, and retry (bounded).
**Reads** are a single projection `get`. Never replay on read.
**Crash between append and projection write:**
- The projection lags the stream. The next write's `expectedVersion` fails, which triggers catch-up, so the system self-heals.
- Provide `get(key, { consistent: true })` that checks the stream head and catches up first.
- Provide a `catchUp(node?)` maintenance method to repair a subtree.
- `insert` ids are time-sortable (ULID-like). Implement a small generator, with no dependency.
### 5.2 Queries and document indexes
- Filter subset:
- `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`
- `$and`, `$or`, `$not`
- implicit equality and dot paths
- Options: `sort`, `limit`, `skip`, `projection` (fields).
- `find` on a node queries that node's **direct children**.
- Indexes are declared per path pattern (same trie mechanism as 3.3).
- Each index has `fields` (compound allowed); entries go under `~p/<node>/~i/<name>/<enc(field1)>/<enc(field2)>/…/<childKey>`, using `encodeIndexValue`.
- `unique: true` additionally claims `~u/<name>/<enc(values)>` via `setIfAbsent`. Throw at construction if the driver lacks `setIfAbsent`.
- **Planner:**
- Pick the index that covers the most leading equality fields, then at most one range field.
- Use the index order for `sort` when it matches.
- Otherwise do a full scan of children projections, with a one-time dev-mode `console.warn` naming the filter.
## 6. Shared conventions
- Every layer uses the same navigation (`child`, `as`, `keys`) and the same self/child arity rules.
- All multi-result reads are `AsyncIterable` (streamed, cancellable). Never build large arrays internally.
- Errors are typed classes exported from `src/errors.ts`: `ConcurrencyError`, `UniqueConstraintError`, `CapabilityError`, `InvalidKeyError`.
- Public exports go through `src/index.ts`, with subpath exports per layer if convenient.
## 7. Types (`src/types/schema.ts`)
Each node is typed by a single **Spec** that describes its own value and its children by key:
```ts
export type Spec = { value?: unknown; children?: Record<string, Spec> }
export type AnySpec = { value: unknown; children: Record<string, AnySpec> } // untyped default
export type ValueOf<S> = S extends { value: infer V } ? V : never
export type ChildrenOf<S> = S extends { children: infer C } ? C : {}
export type KeyOf<S> = Extract<keyof ChildrenOf<S>, string | number>
export type ChildOf<S, K extends KeyOf<S>> = ChildrenOf<S>[K]
// Helpers
export type Many<V, C extends Record<string, Spec> = {}> = { children: { [id: string]: { value: V; children: C } } }
export type Fields<T> = { [K in keyof T & string]: { value: T[K] } }
```
Node signatures:
```ts
class KVNode<S extends Spec = AnySpec> {
child<K extends KeyOf<S>>(key: K): KVNode<ChildOf<S, K>>
get(): Promise<ValueOf<S> | undefined>
get<K extends KeyOf<S>>(key: K): Promise<ValueOf<ChildOf<S, K>> | undefined>
// set/delete/getMany/etc. follow the same pattern
}
```
Example to cover with `expectTypeOf` tests:
```ts
type User = { username: string; age: number }
type Post = { title: string; createdAt: string }
type Schema = {
children: {
users: Many<User, { posts: Many<Post> }>
profile: { children: { [id: string]: { children: Fields<User> } } } // field-per-key style
}
}
const db = new KVStorage<Schema>(driver)
db.child('users').get('123') // User | undefined
db.child('users').child('123').get() // User | undefined (same key)
db.child('users').child('123').child('posts') // KVNode<Many<Post>>
db.child('users').child('123').child('postz') // ✗ type error
db.child('profile').child('9').get('age') // number | undefined
db.child('profile').child('9').get('agee') // ✗ type error
```
Notes:
- **Named levels** check keys at compile time; **dynamic `[id: string]` levels** accept any string.
- One node's children can't mix an index signature with named keys (a TS restriction), so schemas alternate between dynamic and named levels. That alternation exists only in types; the runtime API is just `child()`.
- `EventStorage` and `DocumentStorage` nodes use the same Spec machinery (the value type is the event type or the document type respectively).
## 8. Performance requirements
- Single-key ops: exactly one driver call (no hidden reads), except indexed writes (one `getMany` before the write) and document writes (projection read plus append plus projection write).
- `*Many` ops: one driver call each (plus index work), never N calls.
- `keys()`: skip-scan, O(direct children) seeks.
- Event head lookup: cached; cold lookup is a single reverse scan with `limit: 1`.
- Document `get`: one driver `get`.
- Add `vitest bench` benchmarks for:
- KV `get`/`set`/`getMany` at 10k/100k keys
- `keys()` on a node with 1k children each having 100 descendants
- event append and read
- document `get`/`update`/indexed `find` vs full-scan `find`
- Record baseline numbers in `DESIGN.md`.
## 9. Milestones
1. **Core:** `path.ts` (escape, unescape, encode, `succ`), `encodeIndexValue`, errors. Property tests for round-trips and order preservation.
2. **KV driver:** `KVDriver`, `BaseKVDriver`, `InMemoryKVDriver`, conformance suite.
3. **KV node:** full API from 3.2, including arity rules, skip-scan `keys`, `deleteTree`, `tree`. Add a model-based property test: random ops vs a reference `Map`.
4. **KV indexes:** pattern trie, atomic and non-atomic write paths, read-repair, range `findBy`.
5. **Events:** `KVEventStorage` with commit keys, `setIfAbsent` and mutex modes, head cache, binary-search `read({ from })`, concurrency tests (simulated interleaving with two storage instances on one driver).
6. **Documents:** `EventSourcedDocumentStorage`, update operators, projections, catch-up, queries, planner, unique indexes, `history` and `getAt`.
7. **Types:** Spec machinery wired through all three layers, plus `expectTypeOf` tests from section 7.
8. **HTTP driver:**
- A `createKVHttpHandler(driver)` (framework-agnostic `Request → Response`) plus an `HttpKVDriver` client.
- Endpoints: `POST /getMany`, `POST /setMany`, `POST /deleteMany`, `POST /batch`, `POST /setIfAbsent`, `GET /scan?gte&lt&limit&reverse&values` (NDJSON stream).
- It must pass the conformance suite end-to-end.
9. **Benchmarks** and a `DESIGN.md` update with results.
10. **(Later, ask first)** `MongoDocumentStorage`:
- Path pattern maps to a collection (e.g. `users/*/posts` → `users.posts`), with a `_parent` field and a compound index `(_parent, _id)`.
- Native indexes map from the same index declarations.
- It should pass the document test suite where semantics allow (no `history`/`getAt`).
## 10. Decisions already made (don't revisit without asking)
- `child()`-only navigation; no collection/doc classes.
- `get()` with no argument reads the node's **own key**. It never assembles the subtree; `tree()` does that explicitly.
- A node's own value and its children are independent keys.
- Self/child dispatch by arity; a passed-but-undefined key throws.
- Indexes are declared on path patterns at construction.
- Events use one key per commit, densely numbered.
- Documents store resolved deltas, and reads hit projections only.
- Keys sort lexicographically.
- Values are JSON-serialisable; byte drivers use a codec.
## 11. Open questions to raise with the owner (don't block on them)
- Order-preserving numeric **keys** (not just index values)?
- Should multi-writer safety be **required** (a hard error without `setIfAbsent`) rather than a documented single-writer fallback?
- The hard-delete story for immutable event logs (compaction or crypto-shredding) for privacy deletion requests.
## 12. Implementation decisions
Decisions taken where the brief was silent. Each is the simplest option consistent with sections 1–10.
### M1 — Core
- **Separator** must be a single printable ASCII character other than `%` and `~`.
- **Empty keys are rejected** (`InvalidKeyError`), as are `NaN`/`±Infinity` number keys. An empty segment
would produce `a//b`, which is ambiguous with the subtree layout.
- **Separator escaping** uses `%` + two uppercase hex digits (`/` → `%2F`, `%` → `%25`, leading `~` → `%7E`).
- **`InvalidKeyError` extends `TypeError`**, so "passed-but-undefined key throws a `TypeError`" (3.2) and
the typed error class (6) are the same throw.
- **Index string encoding deviates from "escaped like segments".** Percent-escaping is not
order-preserving (`"a/"` → `"a%2F"` sorts below `"a&"`, and `"a"` + sep sorts above `"a!"`), which
would break `findBy` range scans. Instead every character the index encoder emits sorts strictly
above the separator: code units `<= succ(sep)` are written as `succ(sep)` followed by
`chr(succ(sep) + 1 + c)`. This is monotone and prefix-safe, so `a < b ⇔ enc(a)+sep+… < enc(b)+sep+…`
exactly (property-tested). Encodings are still separator-free.
- **Type tags and number digits**: tags are the first five digits of the digit alphabet; numbers use
16 digits of `0-9a-f` when the separator sorts below `'0'` (e.g. `/`), otherwise the 16 characters
just above the separator. `-0` is stored as `0`; `NaN` cannot be indexed.
- **Ordering** of strings is by UTF-16 code unit (JS `<`). Drivers that compare UTF-8 bytes agree
except for astral-plane characters vs. U+E000–U+FFFF; noted as a driver-author caveat.
### M2 — KV driver
- **Conformance suite location**: `src/testing/kv-conformance.ts`, exported as `potential-storage/testing`.
It imports `vitest`, which is an _optional peer dependency_; the core entry point never imports it.
- **`InMemoryKVDriver` clones values** on write and read (`structuredClone`, skipped for primitives) so
stored state can't be mutated through shared references, matching serialising drivers. Opt out
with `{ clone: false }`.
- **Scans tolerate concurrent writes**: the in-memory scan remembers the last key yielded and re-seeks
if the map changed while suspended. Read-repair and `deleteTree` rely on this. The conformance suite
requires it of every driver (deleting an already-yielded key mid-scan must not skip keys).
- **`setMany` bulk path**: ≥32 new keys are sorted and merged (O(n + k log k)); fewer are spliced.
- **`BaseKVDriver.deleteRange`** deletes in chunks of 1000 (scan with `limit`, then `deleteMany`,
repeat), so it never deletes from under its own live iterator. It also provides `get`/`set`/`delete`
single-key helpers.
- **No fake `batch`**: `BaseKVDriver` deliberately does not provide a non-atomic `batch`, since stores
use its presence to mean "atomic".
### M3 — KV node
- **Child order is "subtree order", not plain segment order.** The brief's skip-scan assumes a
child's keys are contiguous, but they aren't when a sibling extends a segment with a character
that sorts below the separator (for `/`: space and `!"#$%&'()*+,-.`). With children `a` and `a-1`
the key space is `p/a`, `p/a-1`, `p/a-1/x`, `p/a/x`, so `a`'s descendants sort _after_ `a-1`, and
"yield if different from the previous segment" would emit `a` twice. `keys()`/`entries()` therefore
order children by **escaped segment followed by the separator** (`a-1/` < `a/`), which is the order
of their subtrees in key space. For keys without such characters this is plain lexicographic order.
- Forward listing defers a child seen only through its own key on a small stack (bounded by
segment length) until the scan passes `p/<seg>/`. It costs no extra driver calls beyond the
brief's one seek per child with descendants.
- Reverse listing needs one batched `getMany` probe per new child whose segment contains a
below-separator character, to find shorter siblings that only have an own key.
- `after` is exclusive in the direction of travel and follows the same order.
- Implemented once in `src/kv/children.ts` (`scanChildren`) and reused by the event and document
layers. A model-based property test checks forward, reverse, `after` and `limit` against
brute force with keys like `a`, `a-1`, `a.b`, `a/x`, `%`, `~t`.
- **Open question for the owner:** keep subtree order, or change segment escaping so the key
layout nests exactly. That would mean escaping every below-separator character with an escape
character above the separator, at the cost of the readable percent-encoding.
- **Errors from async methods are rejections.** Every async method (`get`, `set`, …) is an `async`
function, so a bad key yields a rejected promise rather than a synchronous throw. `child()` is sync
and throws synchronously.
- **`set(undefined)` is rejected** (`TypeError`: use `delete()`); `undefined` means "missing" in reads.
- **`deleteTree()` on the root** is allowed and clears the whole store (chunked scan + delete, since
the root range has no upper bound). Subtree deletion is `delete(selfKey)` plus
`deleteRange(prefix, selfKey + succ(sep))`. It is not `[selfKey, …)`, which would also catch
siblings such as `a!`.
- **`tree({ depth })`** builds a plain object: leaves are values, interior nodes are objects. A node
with both an own value and children exposes its own value under the `''` property (empty keys are
invalid, so it can't collide). Reserved subtrees are skipped. Deeper levels are skipped by seeking
rather than reading.
- **`keys()` and `entries()` yield string keys** (number keys come back as strings).
- **Exposed low-level properties** on every node: `path`, `selfKey`, `prefix`, `end` (subtree upper
bound), `driver`, `separator`, `isRoot`. The event and document layers build on these.
### M4 — KV indexes
- **Pattern syntax**: segments separated by the storage separator; `*` matches one segment; `''`
names the root. A literal `*` key can't be targeted by a pattern.
- **Overlapping patterns** that declare the same index name throw at construction.
- **Index trie state is stepped per `child()` call** (O(1) in patterns), so each node resolves its
indexes in O(depth) overall. A node holds two bound index sets: one over its children (for child
writes) and its parent's (for self-form writes such as `users.child('1').set(v)`).
- **Index function results**: `undefined` → not indexed; arrays are multi-entry and de-duplicated
per child; anything that is not `string | number | boolean | null` (or `NaN`) throws a `TypeError`
at write time. Index functions are not called for missing (`undefined`) values.
- **`findBy` results** are `[key, value]` pairs (the value is re-read anyway for verification).
Range queries accept `gt`/`gte`/`lt`/`lte`. Results are in index-value order, then child-key order,
and range queries de-duplicate multi-entry hits.
- **`findBy` batching**: entry keys are read in chunks of 128 (or of the remaining `limit`, if
smaller) and values fetched with one `getMany` per chunk. Stale entries are deleted with one
`deleteMany` per chunk.
- **Read-repair race (non-atomic drivers only)**: a reader can see a concurrent writer's fresh
index add before that writer's value write lands, judge it stale, and delete it. With multiple
writers on a driver without `batch`, this can lose an index entry. Mitigation: `readRepair: false`
on `KVStorage` (stale entries are then only skipped, never deleted). With `batch`, entries never go
stale, so the race doesn't arise.
- **`set` of an indexed child with duplicate keys in one `setMany`**: last write wins, and index
diffs are computed once per key.
### M5 — Events
- **`EventStorage` is an interface (type), not a class.** The brief's sample says `new EventStorage(kv)`
but also says it is the abstract interface; the concrete class is `KVEventStorage(kv, opts?)`, which
accepts a `KVStorage` or any `KVNode` as root. Options: `headCacheSize` (default 10 000),
`maxRetries` (default 5) and `now` (clock, default `Date.now`).
- **Commit value**: `{ firstVersion, at, events }`, with `at` in ms since epoch. Read results are
`{ version, commit, at, data }`.
- **The root node is not a stream** (consistent with the root having no own KV value). Non-root
nodes are both a stream and a container of child streams; `keys()` lists child streams, and a
node's `~e` subtree is reserved, so streams and children never mix.
- **`version()` always reads storage** (one reverse scan with `limit: 1`) and refreshes the head cache.
Only `append` trusts the cache. When a cached head disagrees with a numeric `expectedVersion`,
the head is re-read before throwing, so a stale cache never produces a false `ConcurrencyError`.
- **`append([])`** writes nothing; it still checks `expectedVersion` against a fresh head and returns it.
- **Read arity**: `read()` and `read(opts)` are the self form (the first argument is a non-null
object). Any other first argument is a key, so `read(undefined)` throws like every other
passed-but-undefined key.
- **Locating `from`/`to`**: commit `n` has `firstVersion >= n`, so the commit containing version `v` has
`seq <= v`. `read` probes `seq = v` first; with single-event commits that is the answer, in one
`getMany`. Otherwise it runs an 8-ary search, one `getMany` of up to 8 probes per round, which is
O(log₈ commits) round trips. The head is only read when the guess is past the end.
- **Mutex mode** (driver without `setIfAbsent`): a per-stream in-process promise chain serialises
appends. It is safe only with a single writer process.
### M6 — Documents
- **Projection value** is `{ state, version, commit }`, or `{ deleted: true, version, commit }` for a
tombstone. Tombstones keep the version, so re-creating a document continues its stream.
- **`expectedCommit` append hint.** Writes pass the projection's `commit` to `append` as
`expectedCommit`, so a write is exactly projection `getMany` + `setIfAbsent` + projection `batch`
(3 driver calls), even with a cold head cache. A hint taken from a projection can only be stale
(which causes a collision, then catch-up and retry), never ahead of the stream, so it cannot
create a gap. Non-KV backends may ignore it.
- **Per-document in-process write lock.** Writes to the same document within one storage instance
are serialised (`KeyedMutex`), so optimistic retries (default 5) are spent only on cross-process
contention.
- **Return values**: `set` → `void`; `insert` → id; `update` → new state, or `undefined` if the document
doesn't exist (pass `{ upsert: true }` to treat missing as `{}`); `delete` → whether a document
was deleted. No-op updates (resulting state equal to current) append nothing.
- **Arity for documents**: in forms with an optional leading key (`get`, `set`, `update`), a non-null
object first argument selects the self form, and any other first argument is a key (so `undefined`
throws). `delete`/`history` use argument count. `getAt(version)` vs `getAt(key, version)` also uses
argument count, because a version and a numeric key are both numbers.
- **The root document node has no own document** (consistent with KV).
- **`keys()` on a document node** lists children holding a live document. Unlike KV `keys()`, it skips
children that only have nested documents or tombstones.
- **`find` results** are `{ key, data }`. `findOne` returns the same shape or `undefined`.
- **Filter semantics vs MongoDB** (see `src/documents/filter.ts`):
- Comparisons are type-bracketed (`$gt: 5` matches only numbers), and cross-type order follows
the index encoding: `null/missing < false < true < numbers < strings < other`.
- On array fields, every positive operator in one condition (`$eq $gt $gte $lt $lte $in`) must be
satisfied by the **same element** (like `$elemMatch`). In MongoDB, `{ s: { $gt: 5, $lt: 10 } }`
matches `[1, 20]`; here it doesn't. This makes index range scans exact. To get MongoDB's
behaviour, split the condition with `$and`.
- `{ f: null }` matches missing fields. `$not` works both at top level (negating a filter) and at
field level. Unknown operators throw.
- `$in` is not index-accelerated in v1.
- **Index entries** for each field value: scalars as-is, arrays as their scalar elements (multikey;
empty arrays → `null`), and missing or non-scalar values as `null`. Every live document therefore
has at least one entry per index, so a full index scan sees all documents. Indexing a document
where more than one field of a compound index has multiple values throws (as MongoDB does).
- **Index names** default to the fields joined by `,`.
- **Unique indexes**: claims are scoped **per parent node** (`<node>/~u/<name>/<values>`), like one
MongoDB collection per parent. Value tuples where every field is missing/null are not claimed
(sparse). Claims are taken before appending, then released if the append fails. A claim value is
`[ownerKey, claimedAtMs]`. A claim held by another document is taken over only if that
document, after catching up from its stream, no longer holds the value, **and** the claim is older
than `claimGraceMs` (default 30 s). The owner might otherwise be between claiming and appending.
The residual risk is a writer stalling for longer than the grace period between claiming and
appending.
- **Planner**: per index, count leading equality fields (k), then whether field k has a range;
score = 2k + (range ? 1 : 0). Ties prefer an index whose order satisfies `sort` after dropping
equality-constrained sort fields. With no usable predicate, an index matching `sort` is still used
as an ordered full index scan, which streams and stops early with `limit`. Range bounds are clamped
to the bound's type class. Index-backed results are re-read from projections, re-checked against the
full filter, de-duplicated, and stale entries are read-repaired.
- **Result order** when `sort` is not given is the plan's order (index order, or child-key order for
a full scan). Ties in sorted results are broken by child key in the direction of the first sort
field, both for index-ordered and in-memory sorts.
- **In-memory sort** keeps a bounded top `skip + limit` buffer when a limit is given. Without a limit,
it buffers all matches; this is the one place results are materialised.
- **Full-scan warning**: `console.warn` once per filter _shape_ (values replaced by `?`) per storage
instance, only for non-empty filters. On by default unless `NODE_ENV === 'production'`; set it with
`warnOnFullScan`.
- **Catch-up** re-asserts unique claims for the caught-up state with plain writes, since the log is
the source of truth. A slow catch-up can briefly overwrite a newer projection with an older one;
the next write's `expectedVersion` check detects this and heals it. `catchUp()` walks the subtree
via event-stream `keys()` and returns the number of documents repaired.
- **Projections for non-KV event stores** must be passed explicitly (`projections: KVNode`); the
projection key of a document is then `projections.prefix + docKey`.
### M7 — Types
- **`AnySpec` uses an index signature** (`{ [key: string]: AnySpec }`), not `Record<string, …>`, so
`KeyOf<AnySpec>` is `string | number` and number keys type-check on untyped stores.
- **`ChildOf` guards with `infer C extends Spec`** so that `KVNode<ChildOf<S, K>>` satisfies the `Spec`
constraint generically.
- **`setMany` entries are a union of precise `[key, value]` pairs** (`EntryOf<S>`), so on named levels a
value must match its own key (`['age', 'x']` is rejected).
- **Listings**: `keys()` yields `` `${KeyOf<S>}` `` (string literals on named levels, `string` on dynamic
ones). `entries()` and `findBy()` yield `[string, V]` on dynamic levels.
- **"Accept any node" parameters** are typed `KVNode<AnySpec>` / `EventNode<AnySpec>`. A typed node is
assignable to these, but not to `KVNode<any>`, because `KeyOf<any>` collapses to `never`.
- **Type tests** live in `test/types.test-d.ts` and are enforced by `tsc --noEmit` (part of `npm test`),
including the `@ts-expect-error` negative cases from section 7.
### M8 — HTTP driver
- **Extra endpoints** beyond the brief's list: `POST /deleteRange {gte, lt}` and
`GET /capabilities`. Unsupported optional operations return `501`, driver errors `500`, and unknown
routes `404`.
- **Missing vs `null`**: `/getMany` returns `{ values, missing }`, where `missing` lists the indices of
absent keys, because `null` is a legitimate stored value.
- **Codec scope**: the codec serialises whole request/response bodies and each NDJSON scan line
(`[key]` or `[key, value]`), so it must not emit newlines. The default is JSON.
- **Capabilities on the client**: `HttpKVDriver` defines `setIfAbsent`/`batch` only when the server
supports them, so stores' feature detection (e.g. unique indexes requiring `setIfAbsent`, atomic
index writes using `batch`) behaves as it would locally. `HttpKVDriver.connect()` discovers them via
`/capabilities`. The constructor takes them explicitly (default: all). Without native
`deleteRange` the client falls back to `BaseKVDriver`'s chunked delete.
- **Scan streaming**: the server pulls the driver iterator in chunks that grow from 16 to 256
entries, so short reads (skip-scan re-seeks) don't make it run far ahead. Cancelling the client
iterator cancels the response body, which calls `return()` on the server-side iterator.
- **Optional `headers`** (e.g. auth) and an injectable `fetch` on the client; an optional `basePath` on
the handler.
- The conformance suite runs end-to-end in-process (fetch → handler), over a real `node:http`
socket, and against a minimal server driver with discovered capabilities. A smoke test runs KV
indexes, events and unique-indexed documents over HTTP.
### M9 — Benchmarks and in-memory driver performance
- **The in-memory driver's sorted array was replaced by `ChunkedSortedMap`** (sorted chunks of ≤1024
keys, split in half on overflow; values in a `Map`). Profiling showed 59% of document-update time
in the sorted array's O(n) `splice` once the store held ~70k keys (each write adds commit, index
and claim keys). Inserts and deletes are now O(log n + 1024). Iteration is O(1) per step and
re-seeks after a mutation, which replaces the old version-counter revalidation in the driver.
The structure stays behind the `SortedMap` interface; a B-tree remains possible.
- **Value cloning uses a hand-written JSON deep copy** instead of `structuredClone` (about 10× cheaper
per value), in the in-memory driver and the document layer. Values are JSON by contract (§10).
<!-- decisions appended per milestone -->
## 13. Benchmark baselines
`npm run bench` (Vitest bench), in-memory driver with default cloning, Node 22.22, Linux x64 cloud
container (shared CPU, so expect ±10–20% run-to-run). Mean time per operation.
**KV** (`bench/kv.bench.ts`): values are small objects.
| Operation | 10k keys | 100k keys |
| -------------------- | -------- | --------- |
| `get` | 1.2 µs | 1.6 µs |
| `set` (overwrite) | 1.7 µs | 2.0 µs |
| `getMany` × 100 keys | 21 µs | 22 µs |
**`keys()`** (`bench/keys.bench.ts`): node with 1 000 children, each with 100 descendants (101k keys).
| Operation | Mean |
| -------------------------------- | ------ |
| `keys()` skip-scan (1 000 seeks) | 2.9 ms |
| Reference: full subtree scan | 36 ms |
**Events** (`bench/events.bench.ts`): single-event commits.
| Operation | Mean |
| ---------------------------------------------- | ------ |
| `append` 1 event (warm head cache) | 6.3 µs |
| `read` 100 events from version 5 000 of 10 000 | 174 µs |
| `read` full 10 000-event stream | 21 ms |
| `version()` (reverse scan, `limit: 1`) | 3.9 µs |
**Documents** (`bench/documents.bench.ts`): 10 000 docs, indexes on `age` and unique `email`.
| Operation | Mean |
| ----------------------------------------------------------------- | ------- |
| `get` | 2.6 µs |
| `update` (`$inc`, indexed node: projection read + append + batch) | 22 µs |
| `findOne` by unique index (`email`) | 16 µs |
| `find` indexed (`age = k`, ~100 hits) | 0.57 ms |
| `find` full scan (`city = k`, ~200 hits of 10k) | 28 ms |
For comparison, the first run with the sorted-array map and `structuredClone` measured: KV `get`
3.7 µs, `getMany`×100 186 µs, `append` 34 µs, document `update` 110 µs, full scan 57 ms, and
full stream read 62 ms.
## 3. Keys (`utils/keys.ts`)
- A node's key is its escaped path segments joined by `/` (`users/123`). The separator is fixed.
- **Escaping:** every character at or below `/` (space, `!"#$%&'()*+,-.`, `/`, control characters)
and `~` itself becomes `~XX` (uppercase hex). Escaped segments contain only characters above `/`,
so all keys of one child sort together (`p/a`, `p/a/…`, then `p/a~2D1` for `a-1`). That keeps
child listing a plain skip-scan.
- **Reserved segments** are `~` + a lowercase letter (`~e` = event commits). Escaping only produces
`~` + uppercase hex, so user keys can't collide.
- Keys sort lexicographically by escaped form (`10` before `9`; `a-1` → `a~2D1` after `a0`).
- Empty keys, `NaN`/`Infinity` and non-string/number keys are rejected.
## 4. KV storage
`BaseKVStorage` primitives (absolute keys):
| Primitive | |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `readMany(keys)` | values in order, `undefined` for missing |
| `writeMany(entries)` | |
| `removeMany(keys)` | |
| `writeIfAbsent(entries)` | `true` = written; atomic per key (needed by events) |
| `scan({ gte, lt, limit, reverse, values })` | key-ordered; must keep working if keys change mid-scan (keys present throughout appear exactly once) |
Node API (defined once): `child`, `as`, `get(key?)`, `getMany`, `set(value)` / `set(key, value)`,
`setMany`, `delete(key?)`, `deleteMany`, `deleteTree(key?)`, `keys({ limit, after, reverse })`,
`entries(...)`. The root has no own value. A node's own value and its children are independent keys.
- **`keys()`** is a skip-scan: one seek per child that has descendants; siblings without
descendants share one iterator. It lists children that only have descendants, and skips
reserved segments.
- **The only KV "index" is the ordered key space.** There is no `find`: with KV you know where to look,
or you list a node's children.
- **Implementations:** `KVStorageInMemory` (`SortedMap`: sorted chunks of ≤1024 keys; values deep-copied);
`KVStorageHttp` (client) + `KVStorageHttpServer` (wraps any KV storage; `handle(Request) → Response`,
`POST /<primitive>` with JSON bodies, `scan` streamed as NDJSON). The server exposes the whole
underlying store (primitives use absolute keys).
- **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives.
Keys must stay plaintext (or be deterministically encrypted) for prefix scans to work; values can be
encrypted freely.
## 5. Event storage
`BaseEventStorage` primitives over stream paths: `appendTo`, `readFrom`, `versionOf`, `streamsOf`.
Node API: `append(events, opts?)` / `append(key, events, opts?)`, `read(opts?)` / `read(key, opts?)`,
`version(key?)`, `keys()`. The root is not a stream. An empty stream is version 0; the first event is 1.
`expectedVersion: number | 'any'` (default `'any'`).
**`EventStorageKV`** (on any KV storage):
- One key per append, densely numbered: `<stream>/~e/<commit:12>` → `{ firstVersion, at, events }`.
Keying by commit number (not version) means any writer with a stale view collides with the
existing key.
- Append: head from a bounded cache (else one reverse scan, `limit: 1`) → check `expectedVersion`
(re-reading the head before rejecting, in case the cache is stale) → `writeIfAbsent`. On a lost race:
`ConcurrencyError` if a version was expected, otherwise retry (default 5).
- `read({ from })`: commit `n` starts at a version ≥ n, so try commit `from` first (exact for
single-event commits), else binary-search; then forward-scan.
- Multi-writer safe on any store, because `writeIfAbsent` is a required primitive.
## 6. Document storage
`BaseDocumentStorage` primitives over document paths:
| Primitive | |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| `readMany(paths)` | |
| `mutate(path, change)` | maps current state → `DocChange` (or `null` = no-op), applies it, returns the new state; the implementation decides how it's made safe |
| `query(parent, DocumentQuery)` | children of `parent`, filtered, sorted and paged |
Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)` → ULID-like id,
`update(spec, opts?)` / `update(key, spec, opts?)` → new state or `undefined` (`{ upsert }`),
`delete(key?)` → existed, `find(filter, { sort, limit, skip })`, `findOne`, `keys()`.
- **Queries** (`utils/document-query.ts`): `{ field: value }` (dot paths; `null` also matches missing),
`$eq $gt $gte $lt $lte $in`. Comparisons are type-bracketed. On arrays a condition matches the array
or any one element. Sorting order: missing/null < booleans < numbers < strings < other.
- **Updates** (`utils/document-update.ts`): `$set $unset $inc $push $pull $addToSet` (`$each` for
push/addToSet). They resolve to a `DocChange` with final values (`$inc` → `set: { n: 6 }`), so
changes replay deterministically.
- **Indexes:** `{ indexes: { users: ['email', 'age'], 'users/*/posts': ['createdAt'] } }`, passed to
the implementation.
**Implementations**
- `DocumentStorageInMemory`: maps per collection; equality index per declared field (used for
`{ f: v }` / `{ f: { $eq: v } }`), otherwise scans. Writes are atomic in-process.
- `DocumentStorageKV`: each document is the KV value at its path; queries scan the collection's
children; no indexes; concurrent writes to one document are last-writer-wins.
- `EventSourcedDocumentStorage(events, projections)`: each document's `DocChange`s are appended to
its own stream. Current state lives in **any** `BaseDocumentStorage` used as the projection store,
which serves all reads, queries and indexes.
- Projections carry `_version` (reserved; stripped from results), used as `expectedVersion` on the
next append.
- If a write crashes between append and projection update, the next write conflicts, rebuilds the
projection from the stream, and retries. `catchUp(key?)` rebuilds on demand.
- Missing projections with stream history (deleted documents, crashed first writes) are rebuilt
before writing.
- Writes to one document are serialised in-process, so retries only happen across processes.
- Extras: `history(key?)`, `getAt(version)` / `getAt(key, version)`. Children keep these methods
in their types through interface declaration merging (types only).
## 7. Changes from the original brief
- Drivers + node classes → one abstract class per layer (client and server share an API).
- KV indexes (`findBy`, pattern trie, order-preserving value encoding, read-repair) removed.
Document indexes are declared as fields and handled by each store; unique indexes and the query
planner were removed.
- Queries: `$ne $nin $exists $and $or $not` and projections dropped.
- Separator fixed to `/`, and every character ≤ `/` is escaped so child keys stay contiguous. This
removed the subtree-order workaround in child listing.
- Event-sourced documents are events + a projection store instead of a bespoke projection format.
- `writeIfAbsent` is a required primitive (no single-writer mutex fallback; no optional `batch` /
`deleteRange`).
- `tree()` removed (use `entries()` for field-per-key reads).
## 8. Open questions
- Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The
encryption wrapper could support per-subject keys.
- Order-preserving numeric keys (`9` before `10`) as an opt-in key encoding.
## 9. Benchmark baselines
`npm run bench`, in-memory stores, Node 22, Linux x64 cloud container (shared CPU, ±10–20%). Mean per operation.
| Benchmark | Mean |
| -------------------------------------------------------------------- | --------------- |
| KV `get` / `set` @ 10k keys | 1.5 / 1.7 µs |
| KV `get` / `set` @ 100k keys | 2.0 / 2.3 µs |
| KV `getMany` × 100 | 25–35 µs |
| `keys()` on 1k children × 100 descendants (skip-scan vs full scan) | 2.8 ms vs 34 ms |
| Event `append` (warm cache) | 7.8 µs |
| Event `read` 100 from version 5 000 of 10k | 131 µs |
| Event `read` full 10k stream | 18 ms |
| Documents in-memory @ 10k: `get` / `update` | 1.8 / 6.6 µs |
| Documents in-memory: `find` indexed (~100 hits) / scan (~200 of 10k) | 0.20 / 11 ms |
| Event-sourced @ 10k: `get` / `update` | 1.6 / 17 µs |
| Event-sourced: `find` indexed / scan | 0.38 / 12 ms |
bench/document-storage.bench.ts
+33
View File
@@ -0,0 +1,33 @@
import { bench, describe } from 'vitest'
import { DocumentStorageInMemory } from '../src/document-storage/document-storage-in-memory.js'
import { EventSourcedDocumentStorage } from '../src/document-storage/event-sourced-document-storage.js'
import { EventStorageKV } from '../src/event-storage/event-storage-kv.js'
import { KVStorageInMemory } from '../src/kv-storage/kv-storage-in-memory.js'
import type { BaseDocumentStorage } from '../src/document-storage/base-document-storage.js'
const N = 10_000
const stores: [string, BaseDocumentStorage][] = [
['in-memory', new DocumentStorageInMemory({ indexes: { users: ['age'] } })],
['event-sourced', new EventSourcedDocumentStorage(new EventStorageKV(new KVStorageInMemory()), new DocumentStorageInMemory({ indexes: { users: ['age'] } }))],
]
for (const [name, docs] of stores) {
const users = docs.child('users')
for (let i = 0; i < N; i++) await users.set(`u${i}`, { name: `user ${i}`, age: i % 100, city: `c${i % 50}` })
let i = 0
describe(`documents (${name}) @ 10k`, () => {
bench('get', async () => {
await users.get(`u${i++ % N}`)
})
bench('update ($inc)', async () => {
await users.update(`u${i++ % N}`, { $inc: { logins: 1 } })
})
bench('find by indexed field (~100 hits)', async () => {
for await (const _ of users.find({ age: i++ % 100 })) {
}
})
bench('find by unindexed field (scan, ~200 hits)', async () => {
for await (const _ of users.find({ city: `c${i++ % 50}` })) {
}
})
})
}
bench/documents.bench.ts
-34
View File
@@ -1,34 +0,0 @@
import { bench, describe } from 'vitest'
import { InMemoryKVDriver } from '../src/kv/memory.js'
import { KVStorage } from '../src/kv/node.js'
import { KVEventStorage } from '../src/events/kv.js'
import { EventSourcedDocumentStorage } from '../src/documents/event-sourced.js'
const N = 10_000
const docs = new EventSourcedDocumentStorage(new KVEventStorage(new KVStorage(new InMemoryKVDriver())), {
indexes: { users: [{ fields: ['age'] }, { fields: ['email'], unique: true }] },
warnOnFullScan: false,
})
const users = docs.child('users')
for (let i = 0; i < N; i++) await users.set(`u${i}`, { name: `user ${i}`, email: `u${i}@x.io`, age: i % 100, city: `c${i % 50}` })
let i = 0
describe(`documents @ ${N / 1000}k docs`, () => {
bench('get', async () => {
await users.get(`u${i++ % N}`)
})
bench('update ($inc, indexed node)', async () => {
await users.update(`u${i++ % N}`, { $inc: { logins: 1 } })
})
bench('findOne by unique index (email)', async () => {
await users.findOne({ email: `u${i++ % N}@x.io` })
})
bench('find indexed (age = k, ~100 hits)', async () => {
for await (const _ of users.find({ age: i++ % 100 })) {
}
})
bench('find full scan (city = k, ~200 hits)', async () => {
for await (const _ of users.find({ city: `c${i++ % 50}` })) {
}
})
})
bench/event-storage.bench.ts
+22
View File
@@ -0,0 +1,22 @@
import { bench, describe } from 'vitest'
import { EventStorageKV } from '../src/event-storage/event-storage-kv.js'
import { KVStorageInMemory } from '../src/kv-storage/kv-storage-in-memory.js'
const events = new EventStorageKV(new KVStorageInMemory())
const streams = events.child('streams')
for (let i = 0; i < 10_000; i++) await streams.append('long', [{ i }])
let s = 0
describe('events', () => {
bench('append 1 event (warm cache)', async () => {
await streams.append(`s${s++ % 1000}`, [{ type: 'e' }])
})
bench('read 100 events from version 5000 of 10k', async () => {
for await (const _ of streams.read('long', { from: 5000, limit: 100 })) {
}
})
bench('read full 10k-event stream', async () => {
for await (const _ of streams.read('long')) {
}
})
})
bench/events.bench.ts
-27
View File
@@ -1,27 +0,0 @@
import { bench, describe } from 'vitest'
import { InMemoryKVDriver } from '../src/kv/memory.js'
import { KVStorage } from '../src/kv/node.js'
import { KVEventStorage } from '../src/events/kv.js'
const events = new KVEventStorage(new KVStorage(new InMemoryKVDriver()))
const streams = events.child('streams')
const long = streams.child('long')
for (let i = 0; i < 10_000; i++) await long.append([{ type: 'tick', i }])
let s = 0
describe('events', () => {
bench('append 1 event (warm head cache)', async () => {
await streams.append(`s${s++ % 1000}`, [{ type: 'e' }])
})
bench('read 100 events from version 5000 of 10k', async () => {
for await (const _ of long.read({ from: 5000, limit: 100 })) {
}
})
bench('read full 10k-event stream', async () => {
for await (const _ of long.read()) {
}
})
bench('version() (cold head lookup)', async () => {
await long.version()
})
})
bench/keys.bench.ts
-26
View File
@@ -1,26 +0,0 @@
import { bench, describe } from 'vitest'
import { InMemoryKVDriver } from '../src/kv/memory.js'
import { KVStorage } from '../src/kv/node.js'
// 1k children, each with 100 descendants (100k keys under the node).
const db = new KVStorage(new InMemoryKVDriver({ clone: false }))
const node = db.child('n')
const writes: [string, number][] = []
for (let c = 0; c < 1000; c++) {
writes.push([`n/c${c}`, c])
for (let d = 0; d < 100; d++) writes.push([`n/c${c}/d${d}`, d])
}
await db.driver.setMany(writes)
describe('keys() on 1k children x 100 descendants', () => {
bench('keys() skip-scan', async () => {
let n = 0
for await (const _ of node.keys()) n++
if (n !== 1000) throw new Error(`expected 1000, got ${n}`)
})
bench('naive full subtree scan (reference)', async () => {
let n = 0
for await (const _ of db.driver.scan({ gte: 'n/', lt: 'n0' })) n++
if (n !== 101_000) throw new Error(`expected 101000, got ${n}`)
})
})
bench/kv-storage.bench.ts
+40
View File
@@ -0,0 +1,40 @@
import { bench, describe } from 'vitest'
import { KVStorageInMemory } from '../src/kv-storage/kv-storage-in-memory.js'
for (const n of [10_000, 100_000]) {
const db = new KVStorageInMemory()
await db.writeMany(Array.from({ length: n }, (_, i) => [`users/u${i}`, { name: `user ${i}`, n: i }]))
const users = db.child('users')
const keys = Array.from({ length: 100 }, (_, i) => `u${(i * 7919) % n}`)
let i = 0
describe(`KV @ ${n / 1000}k keys`, () => {
bench('get', async () => {
await users.get(`u${i++ % n}`)
})
bench('set', async () => {
await users.set(`u${i++ % n}`, { name: 'x', n: i })
})
bench('getMany x100', async () => {
await users.getMany(keys)
})
})
}
const tree = new KVStorageInMemory()
const writes: [string, number][] = []
for (let c = 0; c < 1000; c++) {
writes.push([`n/c${c}`, c])
for (let d = 0; d < 100; d++) writes.push([`n/c${c}/d${d}`, d])
}
await tree.writeMany(writes)
describe('keys() on 1k children x 100 descendants', () => {
bench('keys() skip-scan', async () => {
for await (const _ of tree.child('n').keys()) {
}
})
bench('full subtree scan (reference)', async () => {
for await (const _ of tree.scan({ gte: 'n/', lt: 'n0' })) {
}
})
})
bench/kv.bench.ts
-29
View File
@@ -1,29 +0,0 @@
import { bench, describe } from 'vitest'
import { InMemoryKVDriver } from '../src/kv/memory.js'
import { KVStorage } from '../src/kv/node.js'
async function populated(n: number) {
const db = new KVStorage(new InMemoryKVDriver())
const users = db.child('users')
const batch: [string, { name: string; n: number }][] = []
for (let i = 0; i < n; i++) batch.push([`u${i}`, { name: `user ${i}`, n: i }])
await users.setMany(batch)
return users
}
for (const n of [10_000, 100_000]) {
const users = await populated(n)
const keys = Array.from({ length: 100 }, (_, i) => `u${(i * 7919) % n}`)
let i = 0
describe(`KV @ ${n / 1000}k keys`, () => {
bench('get', async () => {
await users.get(`u${i++ % n}`)
})
bench('set (overwrite)', async () => {
await users.set(`u${i++ % n}`, { name: 'x', n: i })
})
bench('getMany x100', async () => {
await users.getMany(keys)
})
})
}
package.json
-28
View File
@@ -12,26 +12,6 @@
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
},
"./kv": {
"types": "./dist/kv/index.d.ts",
"import": "./dist/kv/index.js"
},
"./events": {
"types": "./dist/events/index.d.ts",
"import": "./dist/events/index.js"
},
"./documents": {
"types": "./dist/documents/index.d.ts",
"import": "./dist/documents/index.js"
},
"./http": {
"types": "./dist/http/index.d.ts",
"import": "./dist/http/index.js"
},
"./testing": {
"types": "./dist/testing/index.d.ts",
"import": "./dist/testing/index.js"
}
},
"files": [
@@ -53,13 +33,5 @@
"prettier": "^3.9.10",
"typescript": "^5.9.3",
"vitest": "^3.2.7"
},
"peerDependencies": {
"vitest": ">=3"
},
"peerDependenciesMeta": {
"vitest": {
"optional": true
}
}
}
src/core/encoding.ts
-176
View File
@@ -1,176 +0,0 @@
import { DEFAULT_SEPARATOR, succ } from './path.js'
export type IndexValue = string | number | boolean | null
/**
* Order-preserving, separator-safe encoding of index values.
*
* Every character the encoder emits sorts strictly above the separator. That makes encodings
* prefix-safe inside keys: `enc(a) + sep + …` sorts below `enc(b)` whenever `enc(a)` is a proper
* prefix of `enc(b)`, so `a < b ⇔ key(a, …) < key(b, …)` and range scans over index entries are exact.
*
* Layout: one type-tag character, then a body.
* - Type tags order `null < false < true < number < string`.
* - Numbers: IEEE-754 float64, big-endian; positive values have the sign bit flipped, negative
* values have all bits flipped; written as 16 fixed-width "hex" digits. `-0` is normalised to `0`;
* `NaN` is rejected.
* - Strings: each code unit `c` with `c <= E` (where `E = succ(sep)`) is written as `E` followed by
* `chr(E + 1 + c)`; every other code unit is written as-is. This is monotone, so order is kept.
*
* The 16 digits are `0-9a-f` when the separator sorts below `'0'` (e.g. `/`), otherwise the 16
* characters immediately after the separator.
*/
export interface IndexValueCodec {
readonly sep: string
encode(value: IndexValue): string
decode(encoded: string): IndexValue
}
const codecs = new Map<string, IndexValueCodec>()
export function indexValueCodec(sep: string = DEFAULT_SEPARATOR): IndexValueCodec {
let codec = codecs.get(sep)
if (!codec) {
codec = makeCodec(sep)
codecs.set(sep, codec)
}
return codec
}
export function encodeIndexValue(value: IndexValue, sep: string = DEFAULT_SEPARATOR): string {
return indexValueCodec(sep).encode(value)
}
export function decodeIndexValue(encoded: string, sep: string = DEFAULT_SEPARATOR): IndexValue {
return indexValueCodec(sep).decode(encoded)
}
/** Encodes a compound value as `enc(v1) + sep + enc(v2) + …`. */
export function encodeIndexValues(values: readonly IndexValue[], sep: string = DEFAULT_SEPARATOR): string {
const codec = indexValueCodec(sep)
let out = ''
for (let i = 0; i < values.length; i++) {
if (i > 0) out += sep
out += codec.encode(values[i]!)
}
return out
}
export function isIndexValue(v: unknown): v is IndexValue {
return v === null || typeof v === 'string' || typeof v === 'boolean' || (typeof v === 'number' && !Number.isNaN(v))
}
function makeCodec(sep: string): IndexValueCodec {
const sepCode = sep.charCodeAt(0)
const digits = sepCode < 0x30 ? '0123456789abcdef' : Array.from({ length: 16 }, (_, i) => String.fromCharCode(sepCode + 1 + i)).join('')
const digitValue = new Map<string, number>([...digits].map((d, i) => [d, i]))
const TAG_NULL = digits[0]!
const TAG_FALSE = digits[1]!
const TAG_TRUE = digits[2]!
const TAG_NUMBER = digits[3]!
const TAG_STRING = digits[4]!
const esc = succ(sep)
const escCode = esc.charCodeAt(0)
const view = new DataView(new ArrayBuffer(8))
function hex32(n: number): string {
let out = ''
for (let shift = 28; shift >= 0; shift -= 4) out += digits[(n >>> shift) & 0xf]
return out
}
function encodeNumber(n: number): string {
if (Number.isNaN(n)) throw new TypeError('NaN cannot be indexed')
view.setFloat64(0, n === 0 ? 0 : n) // normalise -0
let hi = view.getUint32(0)
let lo = view.getUint32(4)
if (hi & 0x80000000) {
hi = ~hi >>> 0
lo = ~lo >>> 0
} else {
hi = (hi ^ 0x80000000) >>> 0
}
return hex32(hi) + hex32(lo)
}
function decodeNumber(body: string): number {
if (body.length !== 16) throw new TypeError(`Bad encoded number ${JSON.stringify(body)}`)
let hi = 0
let lo = 0
for (let i = 0; i < 8; i++) hi = (hi << 4) | digitValue.get(body[i]!)!
for (let i = 8; i < 16; i++) lo = (lo << 4) | digitValue.get(body[i]!)!
hi >>>= 0
lo >>>= 0
if (hi & 0x80000000) {
hi = (hi ^ 0x80000000) >>> 0
} else {
hi = ~hi >>> 0
lo = ~lo >>> 0
}
view.setUint32(0, hi)
view.setUint32(4, lo)
return view.getFloat64(0)
}
function encodeString(s: string): string {
let out = ''
let start = 0
for (let i = 0; i < s.length; i++) {
const c = s.charCodeAt(i)
if (c <= escCode) {
out += s.slice(start, i) + esc + String.fromCharCode(escCode + 1 + c)
start = i + 1
}
}
return start === 0 ? s : out + s.slice(start)
}
function decodeString(body: string): string {
if (body.indexOf(esc) === -1) return body
let out = ''
for (let i = 0; i < body.length; i++) {
const c = body[i]!
if (c === esc) {
const next = body.charCodeAt(++i)
out += String.fromCharCode(next - escCode - 1)
} else out += c
}
return out
}
return {
sep,
encode(value) {
if (value === null) return TAG_NULL
switch (typeof value) {
case 'boolean':
return value ? TAG_TRUE : TAG_FALSE
case 'number':
return TAG_NUMBER + encodeNumber(value)
case 'string':
return TAG_STRING + encodeString(value)
default:
throw new TypeError(`Cannot index value of type ${typeof value}`)
}
},
decode(encoded) {
const tag = encoded[0]
const body = encoded.slice(1)
switch (tag) {
case TAG_NULL:
return null
case TAG_FALSE:
return false
case TAG_TRUE:
return true
case TAG_NUMBER:
return decodeNumber(body)
case TAG_STRING:
return decodeString(body)
default:
throw new TypeError(`Bad encoded index value ${JSON.stringify(encoded)}`)
}
},
}
}
src/core/lru.ts
-30
View File
@@ -1,30 +0,0 @@
/** Minimal bounded LRU over `Map` insertion order. */
export class LRU<K, V> {
readonly #map = new Map<K, V>()
constructor(readonly capacity: number) {}
get(key: K): V | undefined {
const v = this.#map.get(key)
if (v !== undefined) {
this.#map.delete(key)
this.#map.set(key, v)
}
return v
}
set(key: K, value: V): void {
if (this.capacity <= 0) return
this.#map.delete(key)
this.#map.set(key, value)
if (this.#map.size > this.capacity) this.#map.delete(this.#map.keys().next().value as K)
}
delete(key: K): void {
this.#map.delete(key)
}
get size(): number {
return this.#map.size
}
}
src/core/mutex.ts
-15
View File
@@ -1,15 +0,0 @@
/** Per-key in-process mutex: calls with the same key run one at a time, in order. */
export class KeyedMutex {
readonly #tails = new Map<string, Promise<unknown>>()
run<T>(key: string, fn: () => Promise<T>): Promise<T> {
const prev = this.#tails.get(key) ?? Promise.resolve()
const result = prev.then(fn, fn)
const tail = result.catch(() => {})
this.#tails.set(key, tail)
void tail.then(() => {
if (this.#tails.get(key) === tail) this.#tails.delete(key)
})
return result
}
}
src/core/path.ts
-109
View File
@@ -1,109 +0,0 @@
import { InvalidKeyError } from '../errors.js'
export const DEFAULT_SEPARATOR = '/'
/** Reserved segments. User segments can never start with `~` (a leading `~` is escaped). */
export const RESERVED = {
index: '~i',
unique: '~u',
events: '~e',
projections: '~p',
} as const
export type Key = string | number
/** The character immediately after `c` (by UTF-16 code unit). */
export function succ(c: string): string {
return String.fromCharCode(c.charCodeAt(0) + 1)
}
export function validateSeparator(sep: string): void {
if (sep.length !== 1) throw new TypeError(`Separator must be a single character, got ${JSON.stringify(sep)}`)
const code = sep.charCodeAt(0)
if (code <= 0x20 || code >= 0x7f || sep === '%' || sep === '~') {
throw new TypeError(`Separator must be printable ASCII other than "%" and "~", got ${JSON.stringify(sep)}`)
}
}
function hex2(code: number): string {
return '%' + code.toString(16).toUpperCase().padStart(2, '0')
}
/** Converts a user key to a segment string, rejecting `undefined`/`null`/empty. */
export function toSegment(key: unknown): string {
if (key === undefined || key === null) throw new InvalidKeyError(`Key must not be ${key}`)
if (typeof key === 'number') {
if (!Number.isFinite(key)) throw new InvalidKeyError(`Numeric key must be finite, got ${key}`)
return String(key)
}
if (typeof key !== 'string') throw new InvalidKeyError(`Key must be a string or number, got ${typeof key}`)
if (key === '') throw new InvalidKeyError('Key must not be empty')
return key
}
/**
* Escapes one path segment: percent-encodes `%`, the separator, and a leading `~`.
* The result never contains a raw separator and never starts with `~`.
*/
export function escapeSegment(segment: string, sep: string = DEFAULT_SEPARATOR): string {
// Fast path: nothing to escape (the overwhelmingly common case).
if (segment.indexOf('%') === -1 && segment.indexOf(sep) === -1 && segment.charCodeAt(0) !== 0x7e) return segment
let out = ''
for (let i = 0; i < segment.length; i++) {
const c = segment[i]!
if (c === '%' || c === sep || (i === 0 && c === '~')) out += hex2(c.charCodeAt(0))
else out += c
}
return out
}
export function unescapeSegment(escaped: string): string {
if (escaped.indexOf('%') === -1) return escaped
return escaped.replace(/%([0-9A-Fa-f]{2})/g, (_, h: string) => String.fromCharCode(parseInt(h, 16)))
}
export function isReservedSegment(escaped: string): boolean {
return escaped.charCodeAt(0) === 0x7e // '~'
}
/** Joins unescaped segments into an escaped key. */
export function joinPath(path: readonly string[], sep: string = DEFAULT_SEPARATOR): string {
let out = ''
for (let i = 0; i < path.length; i++) {
if (i > 0) out += sep
out += escapeSegment(path[i]!, sep)
}
return out
}
/** Splits an escaped key into unescaped segments. */
export function splitPath(key: string, sep: string = DEFAULT_SEPARATOR): string[] {
if (key === '') return []
return key.split(sep).map(unescapeSegment)
}
/**
* Precomputed key strings for a node path.
* - `selfKey`: escaped segments joined by the separator (`''` for the root)
* - `prefix`: `selfKey + sep` (`''` for the root) — every descendant key starts with it
* - `end`: exclusive upper bound of the subtree range (`selfKey + succ(sep)`; `undefined` for the root)
*/
export interface PathKeys {
readonly path: readonly string[]
readonly sep: string
readonly selfKey: string
readonly prefix: string
readonly end: string | undefined
}
export function pathKeys(path: readonly string[], sep: string): PathKeys {
const selfKey = joinPath(path, sep)
const root = path.length === 0
return {
path,
sep,
selfKey,
prefix: root ? '' : selfKey + sep,
end: root ? undefined : selfKey + succ(sep),
}
}
src/core/trie.ts
-82
View File
@@ -1,82 +0,0 @@
/**
* Segment trie over path patterns. `*` matches exactly one segment.
* A node's matching state is computed incrementally: `step(parentState, segment)`,
* so resolving a path is O(depth) and `child()` is O(1) in the number of patterns.
*/
export interface TrieNode<T> {
readonly literal: Map<string, TrieNode<T>>
star: TrieNode<T> | undefined
values: { pattern: string; value: T }[]
}
export type TrieState<T> = readonly TrieNode<T>[]
export class PatternTrie<T> {
readonly root: TrieNode<T> = newNode()
constructor(readonly sep: string) {}
/** `''` names the root node. */
add(pattern: string, value: T): void {
let node = this.root
for (const seg of splitPattern(pattern, this.sep)) {
if (seg === '*') node = node.star ??= newNode()
else {
let next = node.literal.get(seg)
if (!next) node.literal.set(seg, (next = newNode()))
node = next
}
}
node.values.push({ pattern, value })
}
get rootState(): TrieState<T> {
return [this.root]
}
step(state: TrieState<T>, segment: string): TrieState<T> {
if (state.length === 0) return state
if (state.length === 1) {
const n = state[0]!
const lit = n.literal.get(segment)
if (lit && n.star) return [lit, n.star]
return lit ? [lit] : n.star ? [n.star] : EMPTY
}
const out: TrieNode<T>[] = []
for (const n of state) {
const lit = n.literal.get(segment)
if (lit) out.push(lit)
if (n.star) out.push(n.star)
}
return out
}
resolve(path: readonly string[]): TrieState<T> {
let state = this.rootState
for (const seg of path) state = this.step(state, seg)
return state
}
/** Values attached to every pattern matching the state's path. */
static values<T>(state: TrieState<T>): { pattern: string; value: T }[] {
if (state.length === 1) return state[0]!.values
return state.flatMap((n) => n.values)
}
}
const EMPTY: TrieState<never> = []
function newNode<T>(): TrieNode<T> {
return { literal: new Map(), star: undefined, values: [] }
}
export function splitPattern(pattern: string, sep: string): string[] {
return pattern === '' ? [] : pattern.split(sep)
}
/** Whether two patterns can match the same path. */
export function patternsOverlap(a: string, b: string, sep: string): boolean {
const sa = splitPattern(a, sep)
const sb = splitPattern(b, sep)
return sa.length === sb.length && sa.every((s, i) => s === '*' || sb[i] === '*' || s === sb[i])
}
src/document-storage/base-document-storage.ts
+117
View File
@@ -0,0 +1,117 @@
import type { Doc } from '../utils/document-path.js'
import { DocumentQuery, type Filter, type FindOptions, type FindResult } from '../utils/document-query.js'
import { DocumentUpdate, type DocChange, type UpdateSpec } from '../utils/document-update.js'
import { Keys } from '../utils/keys.js'
import type { AnySpec, ChildOf, KeyOf, Spec, ValueAt, ValueOf } from '../utils/schema.js'
import { Ulid } from '../utils/ulid.js'
/**
* Fields to index, per collection path pattern (`*` matches one segment), e.g.
* `{ users: ['email', 'age'], 'users/*\/posts': ['createdAt'] }`. Each implementation decides
* how (or whether) to use them; a native database maps them onto its own indexes.
*/
export type IndexDeclarations = Record<string, string[]>
export interface UpdateOptions {
/** Treat a missing document as `{}`. */
upsert?: boolean
}
/** Document type of a node's children. */
export type ChildDoc<S> = ValueOf<ChildOf<S, KeyOf<S>>>
/**
* Document storage. Every node is a document (its own) and a collection of child documents.
* Implementations provide three primitives over document paths; the user-facing API is
* defined here once.
*/
export abstract class BaseDocumentStorage<S extends Spec = AnySpec> {
/** Unescaped path of this node (`[]` for the root, which has no own document). */
path: readonly string[] = []
protected readonly ids = new Ulid()
abstract readMany(paths: (readonly string[])[]): Promise<(Doc | undefined)[]>
/**
* Maps the current state of one document to a change (`null` = no change), applies it and
* returns the new state. Implementations decide how this is made safe against concurrent writers.
*/
abstract mutate(path: readonly string[], change: (current: Doc | undefined) => DocChange | null): Promise<Doc | undefined>
/** Children of `parent` matching the query, filtered, sorted and paged. */
abstract query(parent: readonly string[], query: DocumentQuery): AsyncIterable<FindResult>
child<K extends KeyOf<S>>(key: K): BaseDocumentStorage<ChildOf<S, K>> {
const node = Object.create(this) as BaseDocumentStorage<ChildOf<S, K>>
node.path = [...this.path, Keys.segment(key)]
return node
}
as<S2 extends Spec>(): BaseDocumentStorage<S2> {
return this as unknown as BaseDocumentStorage<S2>
}
async get<K extends KeyOf<S> | undefined = undefined>(key?: K): Promise<ValueAt<S, K> | undefined> {
const [doc] = await this.readMany([this.pathOf(arguments.length > 0, key)])
return doc as ValueAt<S, K> | undefined
}
async getMany<K extends KeyOf<S>>(keys: readonly K[]): Promise<(ValueOf<ChildOf<S, K>> | undefined)[]> {
return (await this.readMany(keys.map((k) => this.pathOf(true, k)))) as (ValueOf<ChildOf<S, K>> | undefined)[]
}
/** `set(doc)` replaces this document; `set(key, doc)` a child document. */
async set<K extends KeyOf<S>>(...args: [doc: ValueOf<S>] | [key: K, doc: ValueOf<ChildOf<S, K>>]): Promise<void> {
const doc = args[args.length - 1]
if (typeof doc !== 'object' || doc === null || Array.isArray(doc)) throw new TypeError('A document must be a plain object')
await this.mutate(this.pathOf(args.length > 1, args[0]), () => ({ op: 'set', doc: doc as Doc }))
}
/** Adds a child document under a new time-sortable id and returns the id. */
async insert(doc: ChildDoc<S>): Promise<string> {
const id = this.ids.generate()
await this.set(id as KeyOf<S>, doc as never)
return id
}
/**
* `update(spec, options?)` this document; `update(key, spec, options?)` a child.
* Returns the new state, or `undefined` if the document doesn't exist (unless `upsert`).
*/
async update<K extends KeyOf<S>>(
...args: [spec: UpdateSpec, options?: UpdateOptions] | [key: K, spec: UpdateSpec, options?: UpdateOptions]
): Promise<ValueAt<S, K> | undefined> {
const hasKey = typeof args[0] !== 'object' || args[0] === null
const spec = args[hasKey ? 1 : 0] as UpdateSpec
const options = (args[hasKey ? 2 : 1] ?? {}) as UpdateOptions
const next = await this.mutate(this.pathOf(hasKey, args[0]), (current) => (current || options.upsert ? DocumentUpdate.resolve(current ?? {}, spec) : null))
return next as ValueAt<S, K> | undefined
}
/** Deletes this document (or a child). Returns whether one existed. */
async delete(key?: KeyOf<S>): Promise<boolean> {
let existed = false
await this.mutate(this.pathOf(arguments.length > 0, key), (current) => ((existed = current !== undefined) ? { op: 'delete' } : null))
return existed
}
/** Child documents matching `filter`. */
find(filter: Filter = {}, options: FindOptions = {}): AsyncIterable<FindResult<ChildDoc<S>>> {
return this.query(this.path, new DocumentQuery(filter, options)) as AsyncIterable<FindResult<ChildDoc<S>>>
}
async findOne(filter: Filter = {}, options: FindOptions = {}): Promise<FindResult<ChildDoc<S>> | undefined> {
for await (const result of this.find(filter, { ...options, limit: 1 })) return result
return undefined
}
/** Keys of child documents. */
async *keys(): AsyncGenerator<string> {
for await (const { key } of this.find()) yield key
}
/** Path of a child (`hasKey`) or this node; validates keys. */
protected pathOf(hasKey: boolean, key: unknown): readonly string[] {
if (hasKey) return [...this.path, Keys.segment(key)]
if (this.path.length === 0) throw new TypeError('The root node has no own document')
return this.path
}
}
src/document-storage/document-storage-in-memory.ts
+112
View File
@@ -0,0 +1,112 @@
import type { Doc } from '../utils/document-path.js'
import { DocumentPath } from '../utils/document-path.js'
import type { DocumentQuery, FindResult } from '../utils/document-query.js'
import { DocumentUpdate, type DocChange } from '../utils/document-update.js'
import { Json } from '../utils/json.js'
import { Keys } from '../utils/keys.js'
import type { AnySpec, Spec } from '../utils/schema.js'
import { BaseDocumentStorage, type IndexDeclarations } from './base-document-storage.js'
export interface DocumentStorageInMemoryOptions {
/** Fields with an equality index, per collection pattern. */
indexes?: IndexDeclarations
}
/** Field → JSON-encoded value → child keys. */
type CollectionIndex = Map<string, Map<string, Set<string>>>
/**
* In-memory document storage. Declared index fields get an equality index per collection, used
* when a filter has `{ field: value }` or `{ field: { $eq: value } }`; everything else is a scan.
*/
export class DocumentStorageInMemory<S extends Spec = AnySpec> extends BaseDocumentStorage<S> {
private readonly collections = new Map<string, Map<string, Doc>>()
private readonly indexes = new Map<string, CollectionIndex>()
private readonly declarations: [pattern: string[], fields: string[]][]
constructor(options: DocumentStorageInMemoryOptions = {}) {
super()
this.declarations = Object.entries(options.indexes ?? {}).map(([pattern, fields]) => [pattern === '' ? [] : pattern.split('/'), fields])
}
async readMany(paths: (readonly string[])[]): Promise<(Doc | undefined)[]> {
return paths.map((path) => Json.clone(this.collection(path.slice(0, -1)).get(path.at(-1)!)))
}
async mutate(path: readonly string[], change: (current: Doc | undefined) => DocChange | null): Promise<Doc | undefined> {
const parent = path.slice(0, -1)
const key = path.at(-1)!
const docs = this.collection(parent)
const current = docs.get(key)
const resolved = change(Json.clone(current))
if (!resolved) return Json.clone(current)
const next = DocumentUpdate.apply(current, resolved)
if (next) docs.set(key, next)
else docs.delete(key)
this.reindex(parent, key, current, next)
return Json.clone(next)
}
async *query(parent: readonly string[], query: DocumentQuery): AsyncGenerator<FindResult> {
const docs = this.collection(parent)
const keys = [...(this.lookup(parent, query) ?? docs.keys())].sort()
yield* query.run(
(async function* () {
for (const key of keys) {
const doc = docs.get(key)
if (doc) yield { key, data: Json.clone(doc) }
}
})(),
)
}
private collection(parent: readonly string[]): Map<string, Doc> {
const id = Keys.join(parent)
let docs = this.collections.get(id)
if (!docs) this.collections.set(id, (docs = new Map()))
return docs
}
/** Candidate keys from an equality index, if the filter allows one. */
private lookup(parent: readonly string[], query: DocumentQuery): Iterable<string> | undefined {
const index = this.indexes.get(Keys.join(parent))
for (const field of this.fieldsFor(parent)) {
let value = query.filter[field]
if (Json.isObject(value) && Object.keys(value).length === 1 && '$eq' in value) value = value.$eq
if (value === undefined || typeof value === 'object') continue
return index?.get(field)?.get(JSON.stringify(value)) ?? []
}
return undefined
}
private reindex(parent: readonly string[], key: string, before: Doc | undefined, after: Doc | undefined): void {
const fields = this.fieldsFor(parent)
if (fields.length === 0) return
const id = Keys.join(parent)
let index = this.indexes.get(id)
if (!index) this.indexes.set(id, (index = new Map()))
for (const field of fields) {
let byValue = index.get(field)
if (!byValue) index.set(field, (byValue = new Map()))
for (const value of DocumentStorageInMemory.indexValues(before, field)) byValue.get(value)?.delete(key)
for (const value of DocumentStorageInMemory.indexValues(after, field)) {
let keys = byValue.get(value)
if (!keys) byValue.set(value, (keys = new Set()))
keys.add(key)
}
}
}
private fieldsFor(parent: readonly string[]): string[] {
return this.declarations
.filter(([pattern]) => pattern.length === parent.length && pattern.every((s, i) => s === '*' || s === parent[i]))
.flatMap(([, fields]) => fields)
}
/** Index keys for a field: its value, plus each element of an array (missing indexes as `null`). */
private static indexValues(doc: Doc | undefined, field: string): string[] {
if (!doc) return []
const value = DocumentPath.get(doc, field) ?? null
return Array.isArray(value) ? value.map((v) => JSON.stringify(v)) : [JSON.stringify(value)]
}
}
src/document-storage/document-storage-kv.ts
+46
View File
@@ -0,0 +1,46 @@
import type { BaseKVStorage } from '../kv-storage/base-kv-storage.js'
import type { Doc } from '../utils/document-path.js'
import type { DocumentQuery, FindResult } from '../utils/document-query.js'
import { DocumentUpdate, type DocChange } from '../utils/document-update.js'
import { Keys } from '../utils/keys.js'
import type { AnySpec, Spec } from '../utils/schema.js'
import { BaseDocumentStorage } from './base-document-storage.js'
/**
* Document storage on any KV storage: each document is the value at its path. Queries scan the
* collection's direct children (no indexes). Writes are read-modify-write, so concurrent writers
* to the same document are last-writer-wins; use `EventSourcedDocumentStorage` if that matters.
*/
export class DocumentStorageKV<S extends Spec = AnySpec> extends BaseDocumentStorage<S> {
constructor(private readonly kv: BaseKVStorage) {
super()
}
async readMany(paths: (readonly string[])[]): Promise<(Doc | undefined)[]> {
return (await this.kv.readMany(paths.map((p) => this.keyOf(p)))) as (Doc | undefined)[]
}
async mutate(path: readonly string[], change: (current: Doc | undefined) => DocChange | null): Promise<Doc | undefined> {
const key = this.keyOf(path)
const [current] = (await this.kv.readMany([key])) as (Doc | undefined)[]
const resolved = change(current)
if (!resolved) return current
const next = DocumentUpdate.apply(current, resolved)
if (next) await this.kv.writeMany([[key, next]])
else await this.kv.removeMany([key])
return next
}
async *query(parent: readonly string[], query: DocumentQuery): AsyncGenerator<FindResult> {
const collection = parent.reduce<BaseKVStorage>((node, segment) => node.child(segment), this.kv)
yield* query.run(
(async function* () {
for await (const [key, data] of collection.entries()) yield { key, data: data as Doc }
})(),
)
}
private keyOf(path: readonly string[]): string {
return this.kv.prefix + Keys.join(path)
}
}
src/document-storage/event-sourced-document-storage.ts
+125
View File
@@ -0,0 +1,125 @@
import type { BaseEventStorage, StoredEvent } from '../event-storage/base-event-storage.js'
import type { Doc } from '../utils/document-path.js'
import type { DocumentQuery, FindResult } from '../utils/document-query.js'
import { DocumentUpdate, type DocChange } from '../utils/document-update.js'
import { ConcurrencyError } from '../utils/errors.js'
import type { AnySpec, ChildOf, KeyOf, Spec, ValueAt } from '../utils/schema.js'
import { BaseDocumentStorage } from './base-document-storage.js'
export interface EventSourcedDocumentStorageOptions {
/** Retries after a write conflict, each preceded by a catch-up. Default 5. */
maxRetries?: number
}
// Types only: children of an event-sourced store are event-sourced stores too (child() creates
// them with Object.create, so they share this class's prototype).
export interface EventSourcedDocumentStorage<S extends Spec = AnySpec> {
child<K extends KeyOf<S>>(key: K): EventSourcedDocumentStorage<ChildOf<S, K>>
}
/**
* Documents as event streams: each document's changes are appended to its own stream, and the
* current state is kept in a projection store (any `BaseDocumentStorage`), which serves all
* reads and queries.
*
* Projections carry a `_version` field (stripped from results) used as `expectedVersion` on the
* next append. If a write crashes between appending and updating the projection, the next
* write's append conflicts, the projection is rebuilt from the stream, and the write retries.
* `_version` is therefore reserved in documents.
*/
export class EventSourcedDocumentStorage<S extends Spec = AnySpec> extends BaseDocumentStorage<S> {
private readonly maxRetries: number
/** Tail of the pending writes per document, so writes in one process don't conflict with each other. */
private readonly locks = new Map<string, Promise<unknown>>()
constructor(
private readonly events: BaseEventStorage,
private readonly projections: BaseDocumentStorage,
options: EventSourcedDocumentStorageOptions = {},
) {
super()
this.maxRetries = options.maxRetries ?? 5
}
async readMany(paths: (readonly string[])[]): Promise<(Doc | undefined)[]> {
return (await this.projections.readMany(paths)).map((doc) => EventSourcedDocumentStorage.strip(doc))
}
async *query(parent: readonly string[], query: DocumentQuery): AsyncGenerator<FindResult> {
for await (const { key, data } of this.projections.query(parent, query)) yield { key, data: EventSourcedDocumentStorage.strip(data)! }
}
mutate(path: readonly string[], change: (current: Doc | undefined) => DocChange | null): Promise<Doc | undefined> {
const id = JSON.stringify(path)
const result = (this.locks.get(id) ?? Promise.resolve()).then(() => this.write(path, change))
const tail = result.catch(() => {})
this.locks.set(id, tail)
void tail.then(() => this.locks.get(id) === tail && this.locks.delete(id))
return result
}
private async write(path: readonly string[], change: (current: Doc | undefined) => DocChange | null): Promise<Doc | undefined> {
for (let attempt = 0; ; attempt++) {
const [stored] = await this.projections.readMany([path])
const version = (stored?._version as number | undefined) ?? (await this.events.versionOf(path))
if (!stored && version > 0) {
// No projection but the stream has history (deleted, or a crashed first write): rebuild.
if ((await this.rebuild(path)) !== undefined) continue
}
const current = EventSourcedDocumentStorage.strip(stored)
const resolved = change(current)
if (!resolved) return current
const next = DocumentUpdate.apply(current, resolved)
try {
await this.events.appendTo(path, [resolved], version)
} catch (error) {
if (!(error instanceof ConcurrencyError) || attempt >= this.maxRetries) throw error
await this.rebuild(path)
continue
}
await this.project(path, next, version + 1)
return next
}
}
/** Events of this document (or a child). */
history(key?: KeyOf<S>): AsyncIterable<StoredEvent<DocChange>> {
return this.events.readFrom(this.pathOf(arguments.length > 0, key), {}) as AsyncIterable<StoredEvent<DocChange>>
}
/** State as of `version`: `getAt(version)` for this document, `getAt(key, version)` for a child. */
async getAt<K extends KeyOf<S>>(...args: [version: number] | [key: K, version: number]): Promise<ValueAt<S, K> | undefined> {
const path = this.pathOf(args.length > 1, args[0])
let state: Doc | undefined
for await (const event of this.events.readFrom(path, { to: args[args.length - 1] as number })) {
state = DocumentUpdate.apply(state, event.data as DocChange)
}
return state as ValueAt<S, K> | undefined
}
/** Rebuilds the projection of this document (or a child) from its stream. Returns the state. */
catchUp(key?: KeyOf<S>): Promise<Doc | undefined> {
return this.rebuild(this.pathOf(arguments.length > 0, key))
}
private async rebuild(path: readonly string[]): Promise<Doc | undefined> {
let state: Doc | undefined
let version = 0
for await (const event of this.events.readFrom(path, {})) {
state = DocumentUpdate.apply(state, event.data as DocChange)
version = event.version
}
await this.project(path, state, version)
return state
}
private async project(path: readonly string[], state: Doc | undefined, version: number): Promise<void> {
await this.projections.mutate(path, () => (state ? { op: 'set', doc: { ...state, _version: version } } : { op: 'delete' }))
}
private static strip(doc: Doc | undefined): Doc | undefined {
if (!doc) return undefined
const { _version, ...state } = doc
return state
}
}
src/documents/event-sourced.ts
-606
View File
@@ -1,606 +0,0 @@
import type { IndexValue } from '../core/encoding.js'
import { escapeSegment, RESERVED, succ, toSegment, unescapeSegment } from '../core/path.js'
import type { TrieState } from '../core/trie.js'
import { CapabilityError, ConcurrencyError, UniqueConstraintError } from '../errors.js'
import { KeyedMutex } from '../core/mutex.js'
import { KVEventNode } from '../events/kv.js'
import type { EventNode, ReadOptions, StoredEvent } from '../events/types.js'
import { scanChildren } from '../kv/children.js'
import type { BatchOp, KVDriver } from '../kv/driver.js'
import type { KVNode } from '../kv/node.js'
import type { AnySpec, ChildOf, KeyOf, KeyString, Spec, ValueOf } from '../types/schema.js'
import { analyze, matches, type Filter } from './filter.js'
import { DocIndexes, DocIndexRegistry, type DocIndexDeclarations, type Plan } from './indexes.js'
import type { ChildDoc, DocKeysOptions, EventSourcedDocumentNode, FindOptions, FindResult, GetOptions, UpdateOptions } from './types.js'
import { ulid } from './ulid.js'
import { applyEvent, applyPatch, resolveUpdate, toPatchEvent, type DocEvent, type UpdateSpec } from './update.js'
import { clone, compareValues, deepEqual, getPath, isPlainObject, setPath, sortKey, unsetPath, type Doc } from './values.js'
export interface EventSourcedDocumentStorageOptions {
/** Where projections live. Default: the `~p` root of the event store's KV. Required for non-KV event stores. */
projections?: KVNode<AnySpec>
/** Path pattern → indexes over that node's direct children. */
indexes?: DocIndexDeclarations
/** Delete stale index entries found by `find`. Default `true`. */
readRepair?: boolean
/** Retries after a write conflict (each preceded by a catch-up). Default 5. */
maxRetries?: number
/** Warn once per filter shape when `find` falls back to a full scan. Default: `NODE_ENV !== 'production'`. */
warnOnFullScan?: boolean
/** Id generator for `insert`. Default: ULID-like. */
newId?: () => string
/**
* A unique claim whose owner no longer holds the value is only taken over once it is older
* than this, since its owner may be mid-write. Default 30 000 ms.
*/
claimGraceMs?: number
/** Clock for claim timestamps. Default `Date.now`. */
now?: () => number
}
/** Stored projection of one document. */
export type Projection =
{ state: Doc; version: number; commit: number; deleted?: undefined } | { deleted: true; version: number; commit: number; state?: undefined }
interface DocContext {
readonly driver: KVDriver
readonly sep: string
/** Projection-space prefix: projection key of a doc = `projBase + docSelfKey`. */
readonly projBase: string
readonly registry: DocIndexRegistry
readonly readRepair: boolean
readonly maxRetries: number
readonly warnOnFullScan: boolean
readonly warned: Set<string>
readonly newId: () => string
readonly claimGraceMs: number
readonly now: () => number
/** Serialises writes to one document within this process, so retries only arise across processes. */
readonly writeLock: KeyedMutex
}
/** Stored value of a unique claim: owner child segment (escaped) and claim time. */
type Claim = [owner: string, at: number]
interface Target {
ev: EventNode<AnySpec>
parentEv: EventNode<AnySpec>
esc: string
/** Projection key of the document. */
projKey: string
/** Projection-space prefix of its parent (sibling projection key = `parentProj + esc`). */
parentProj: string
idx: DocIndexes | undefined
}
type Hit = { esc: string; state: Doc }
const FIND_CHUNK = 128
/**
* Document node backed by an event stream per document and a projection per document.
* Writes: read projection → resolve delta → append (expectedVersion = projection.version) →
* write projection + index entries. Reads hit the projection only.
*/
export class EventSourcedDocumentNodeImpl<S extends Spec = AnySpec> implements EventSourcedDocumentNode<S> {
readonly path: readonly string[]
/** Document-space key (escaped), relative to the store root. */
readonly selfKey: string
readonly #ctx: DocContext
readonly #ev: EventNode<AnySpec>
readonly #parentEv: EventNode<AnySpec> | undefined
readonly #esc: string
readonly #prefix: string
readonly #state: TrieState<any>
readonly #selfIdx: DocIndexes | undefined
readonly #childIdx: DocIndexes | undefined
/** @internal Use `EventSourcedDocumentStorage` and `child()`. */
constructor(
ctx: DocContext,
ev: EventNode<AnySpec>,
parentEv: EventNode<AnySpec> | undefined,
path: readonly string[],
selfKey: string,
esc: string,
state: TrieState<any>,
selfIdx: DocIndexes | undefined,
) {
this.#ctx = ctx
this.#ev = ev
this.#parentEv = parentEv
this.path = path
this.selfKey = selfKey
this.#esc = esc
this.#prefix = path.length === 0 ? '' : selfKey + ctx.sep
this.#state = state
this.#selfIdx = selfIdx
this.#childIdx = ctx.registry.bind(state, ctx.projBase + this.#prefix)
}
child<K extends KeyOf<S>>(key: K): EventSourcedDocumentNodeImpl<ChildOf<S, K>> {
const seg = toSegment(key)
const esc = escapeSegment(seg, this.#ctx.sep)
return new EventSourcedDocumentNodeImpl(
this.#ctx,
this.#ev.child(seg),
this.#ev,
[...this.path, seg],
this.#prefix + esc,
esc,
this.#ctx.registry.trie.step(this.#state, seg),
this.#childIdx,
)
}
as<S2 extends Spec>(): EventSourcedDocumentNodeImpl<S2> {
return this as unknown as EventSourcedDocumentNodeImpl<S2>
}
// ---------------------------------------------------------------- reads
get(opts?: GetOptions): Promise<ValueOf<S> | undefined>
get<K extends KeyOf<S>>(key: K, opts?: GetOptions): Promise<ValueOf<ChildOf<S, K>> | undefined>
async get(...args: unknown[]): Promise<unknown> {
const [t, rest] = this.#split(args)
const opts = (rest[0] ?? {}) as GetOptions
let proj = await this.#projection(t)
if (opts.consistent && (await t.ev.version()) > (proj?.version ?? 0)) {
await this.#catchUpDoc(t)
proj = await this.#projection(t)
}
return proj?.state
}
history(): AsyncIterable<StoredEvent<DocEvent>>
history<K extends KeyOf<S>>(key: K): AsyncIterable<StoredEvent<DocEvent>>
history(...args: unknown[]): AsyncIterable<StoredEvent<DocEvent>> {
const t = args.length === 0 ? this.#self() : this.#child(args[0])
return t.ev.read() as AsyncIterable<StoredEvent<DocEvent>>
}
getAt(version: number): Promise<ValueOf<S> | undefined>
getAt<K extends KeyOf<S>>(key: K, version: number): Promise<ValueOf<ChildOf<S, K>> | undefined>
async getAt(...args: unknown[]): Promise<unknown> {
const t = args.length <= 1 ? this.#self() : this.#child(args[0])
const version = args[args.length - 1] as number
if (!Number.isInteger(version) || version < 0) throw new TypeError(`Invalid version ${String(version)}`)
let state: Doc | undefined
for await (const e of t.ev.read({ to: version } satisfies ReadOptions)) state = applyEvent(state, e.data as DocEvent)
return state
}
async *find(filter: Filter = {}, opts: FindOptions = {}): AsyncGenerator<FindResult<ChildDoc<S>>> {
const skip = opts.skip ?? 0
const limit = opts.limit ?? Infinity
if (limit <= 0) return
const sort = Object.entries(opts.sort ?? {}) as [string, 1 | -1][]
const plan = this.#childIdx?.plan(analyze(filter), sort)
if (!plan && Object.keys(filter).length > 0) this.#warnFullScan(filter)
const source = plan ? this.#indexScan(plan, skip + limit) : this.#projectionScan()
const project = opts.projection ? projector(opts.projection) : (d: Doc) => d
if (sort.length && !plan?.sorted) {
// In-memory sort, bounded to skip + limit when a limit is given.
const cap = skip + limit
const cmp = comparator(sort)
const top: Hit[] = []
for await (const hit of source) {
if (!matches(hit.state, filter)) continue
if (top.length >= cap && cmp(hit, top[top.length - 1]!) >= 0) continue
let lo = 0
let hi = top.length
while (lo < hi) {
const mid = (lo + hi) >>> 1
if (cmp(top[mid]!, hit) <= 0) lo = mid + 1
else hi = mid
}
top.splice(lo, 0, hit)
if (top.length > cap) top.pop()
}
for (let i = skip; i < top.length; i++) yield { key: unescapeSegment(top[i]!.esc), data: project(top[i]!.state) as ChildDoc<S> }
return
}
let skipped = 0
let emitted = 0
for await (const hit of source) {
if (!matches(hit.state, filter)) continue
if (skipped < skip) {
skipped++
continue
}
yield { key: unescapeSegment(hit.esc), data: project(hit.state) as ChildDoc<S> }
if (++emitted >= limit) return
}
}
async findOne(filter: Filter = {}, opts: FindOptions = {}): Promise<FindResult<ChildDoc<S>> | undefined> {
for await (const r of this.find(filter, { ...opts, limit: 1 })) return r
return undefined
}
async *keys(opts: DocKeysOptions = {}): AsyncGenerator<KeyString<S>> {
let remaining = opts.limit ?? Infinity
if (remaining <= 0) return
const after = opts.after === undefined ? undefined : toSegment(opts.after)
const { prefix, end } = this.#childRange()
for await (const hit of scanChildren(this.#ctx.driver, prefix, end, this.#ctx.sep, { after, reverse: opts.reverse, values: true })) {
if (!hit.hasOwn || (hit.value as Projection).deleted) continue
yield unescapeSegment(hit.esc) as KeyString<S>
if (--remaining <= 0) return
}
}
// ---------------------------------------------------------------- writes
set(doc: ValueOf<S>): Promise<void>
set<K extends KeyOf<S>>(key: K, doc: ValueOf<ChildOf<S, K>>): Promise<void>
async set(...args: unknown[]): Promise<void> {
const [t, rest] = this.#split(args)
const doc = rest[0]
if (!isPlainObject(doc)) throw new TypeError('A document must be a plain object')
const snapshot = clone(doc)
await this.#write(t, () => ({ op: 'set', doc: snapshot }))
}
async insert(doc: ChildDoc<S>): Promise<string> {
if (!isPlainObject(doc)) throw new TypeError('A document must be a plain object')
const id = this.#ctx.newId()
const t = this.#child(id)
const snapshot = clone(doc)
await this.#write(t, (cur) => {
if (cur) throw new Error(`Document "${id}" already exists`)
return { op: 'set', doc: snapshot }
})
return id
}
update(spec: UpdateSpec, opts?: UpdateOptions): Promise<ValueOf<S> | undefined>
update<K extends KeyOf<S>>(key: K, spec: UpdateSpec, opts?: UpdateOptions): Promise<ValueOf<ChildOf<S, K>> | undefined>
async update(...args: unknown[]): Promise<unknown> {
const [t, rest] = this.#split(args)
const spec = rest[0] as UpdateSpec
const opts = (rest[1] ?? {}) as UpdateOptions
if (!isPlainObject(spec)) throw new TypeError('update() expects an update spec')
const result = await this.#write(t, (cur) => {
if (!cur && !opts.upsert) return null
const base = cur ?? {}
const patch = resolveUpdate(base, spec)
if (cur && deepEqual(applyPatch(base, patch), cur)) return null // no-op: no event
return toPatchEvent(patch)
})
return result
}
delete(): Promise<boolean>
delete<K extends KeyOf<S>>(key: K): Promise<boolean>
async delete(...args: unknown[]): Promise<boolean> {
const t = args.length === 0 ? this.#self() : this.#child(args[0])
let deleted = false
await this.#write(t, (cur) => {
deleted = cur !== undefined
return cur ? { op: 'delete' } : null
})
return deleted
}
// ---------------------------------------------------------------- maintenance
async catchUp(): Promise<number> {
let repaired = 0
if (this.path.length > 0 && (await this.#catchUpDoc(this.#self()))) repaired++
for await (const key of this.#ev.keys()) repaired += await this.child(key as KeyOf<S>).catchUp()
return repaired
}
// ---------------------------------------------------------------- internals
#self(): Target {
if (this.path.length === 0) throw new TypeError('The root node has no own document')
const parentProj = this.#ctx.projBase + this.selfKey.slice(0, this.selfKey.length - this.#esc.length)
return { ev: this.#ev, parentEv: this.#parentEv!, esc: this.#esc, projKey: this.#ctx.projBase + this.selfKey, parentProj, idx: this.#selfIdx }
}
#child(key: unknown): Target {
const seg = toSegment(key)
const esc = escapeSegment(seg, this.#ctx.sep)
const parentProj = this.#ctx.projBase + this.#prefix
return { ev: this.#ev.child(seg), parentEv: this.#ev, esc, projKey: parentProj + esc, parentProj, idx: this.#childIdx }
}
/** Target for a sibling of `t` given its escaped segment. */
#sibling(t: Target, esc: string): Target {
return { ev: t.parentEv.child(unescapeSegment(esc)), parentEv: t.parentEv, esc, projKey: t.parentProj + esc, parentProj: t.parentProj, idx: t.idx }
}
/** Self form when there are no args or the first is a non-null object; otherwise key form. */
#split(args: unknown[]): [Target, unknown[]] {
if (args.length === 0 || (typeof args[0] === 'object' && args[0] !== null)) return [this.#self(), args]
return [this.#child(args[0]), args.slice(1)]
}
#childRange(): { prefix: string; end: string | undefined } {
const prefix = this.#ctx.projBase + this.#prefix
return { prefix, end: prefix === '' ? undefined : prefix.slice(0, -1) + succ(this.#ctx.sep) }
}
async #projection(t: Target): Promise<Projection | undefined> {
return (await this.#ctx.driver.getMany([t.projKey]))[0] as Projection | undefined
}
/**
* The write path. `compute` maps the current state to an event (or `null` for a no-op).
* Returns the resulting state.
*/
#write(t: Target, compute: (cur: Doc | undefined) => DocEvent | null): Promise<Doc | undefined> {
return this.#ctx.writeLock.run(t.projKey, () => this.#writeUnlocked(t, compute))
}
async #writeUnlocked(t: Target, compute: (cur: Doc | undefined) => DocEvent | null): Promise<Doc | undefined> {
const { maxRetries } = this.#ctx
for (let attempt = 0; ; attempt++) {
const proj = await this.#projection(t)
const cur = proj?.state
const event = compute(cur)
if (!event) return cur
const next = applyEvent(cur, event)
const oldClaims = t.idx?.claimsFor(cur)
const newClaims = t.idx?.claimsFor(next)
const acquired: string[] = []
if (newClaims?.size) {
const wanted = new Map([...newClaims].filter(([k]) => !oldClaims!.has(k)))
if (wanted.size) await this.#claim(t, wanted, acquired)
}
let result
try {
result = await t.ev.append([event], { expectedVersion: proj?.version ?? 0, expectedCommit: proj?.commit ?? 0 })
} catch (err) {
if (acquired.length) await this.#ctx.driver.deleteMany(acquired)
if (err instanceof ConcurrencyError && attempt < maxRetries) {
await this.#catchUpDoc(t)
continue
}
throw err
}
const releases = oldClaims ? [...oldClaims.keys()].filter((k) => !newClaims!.has(k)) : []
const projection: Projection = next
? { state: next, version: result.version, commit: result.commit }
: { deleted: true, version: result.version, commit: result.commit }
await this.#writeProjection(t, cur, next, projection, releases)
return next
}
}
/**
* Claims unique values for `t`; records successful claims in `acquired`. Throws
* `UniqueConstraintError` on conflict. A claim held by another document is taken over only if
* that document, caught up to its stream head, no longer holds the value and the claim is older
* than `claimGraceMs` (its owner might otherwise be between claiming and appending).
*/
async #claim(t: Target, claims: Map<string, { index: string; values: IndexValue[] }>, acquired: string[]): Promise<void> {
const { driver, now, claimGraceMs } = this.#ctx
const keys = [...claims.keys()]
const written = await driver.setIfAbsent!(keys.map((k) => [k, [t.esc, now()] satisfies Claim]))
const lost = keys.filter((k, i) => {
if (written[i]) acquired.push(k)
return !written[i]
})
if (!lost.length) return
const holders = (await driver.getMany(lost)) as (Claim | undefined)[]
const fail = async (k: string, owner?: string): Promise<never> => {
if (acquired.length) await driver.deleteMany(acquired.splice(0))
const c = claims.get(k)!
throw new UniqueConstraintError(c.index, c.values, owner === undefined ? undefined : unescapeSegment(owner))
}
for (let i = 0; i < lost.length; i++) {
const k = lost[i]!
const holder = holders[i]
if (holder?.[0] === t.esc) continue // already ours (e.g. left by an interrupted write)
if (holder) {
const owner = this.#sibling(t, holder[0])
await this.#catchUpDoc(owner)
const ownerProj = await this.#projection(owner)
if (t.idx!.claimsFor(ownerProj?.state).has(k)) await fail(k, holder[0])
if (now() - holder[1] < claimGraceMs) await fail(k, holder[0])
await driver.deleteMany([k])
}
const [ok] = await driver.setIfAbsent!([[k, [t.esc, now()] satisfies Claim]])
if (!ok) await fail(k)
acquired.push(k)
}
}
async #writeProjection(t: Target, cur: Doc | undefined, next: Doc | undefined, projection: Projection, releases: string[]): Promise<void> {
const { driver } = this.#ctx
if (!t.idx) {
await driver.setMany([[t.projKey, projection]])
return
}
const before = t.idx.entriesFor(t.esc, cur)
const after = t.idx.entriesFor(t.esc, next)
const adds = [...after].filter((e) => !before.has(e))
const removes = [...before].filter((e) => !after.has(e)).concat(releases)
if (driver.batch) {
const ops: BatchOp[] = adds.map((key) => ({ type: 'set', key, value: true }))
ops.push({ type: 'set', key: t.projKey, value: projection })
for (const key of removes) ops.push({ type: 'delete', key })
await driver.batch(ops)
return
}
if (adds.length) await driver.setMany(adds.map((k) => [k, true]))
await driver.setMany([[t.projKey, projection]])
if (removes.length) await driver.deleteMany(removes)
}
/** Replays events newer than the projection into it. Returns whether anything changed. */
async #catchUpDoc(t: Target): Promise<boolean> {
const proj = await this.#projection(t)
const cur = proj?.state
let state = cur
let version = proj?.version ?? 0
let commit = proj?.commit ?? 0
for await (const e of t.ev.read({ from: version + 1 })) {
state = applyEvent(state, e.data as DocEvent)
version = e.version
commit = e.commit
}
if (version === (proj?.version ?? 0)) return false
const oldClaims = t.idx?.claimsFor(cur)
const newClaims = t.idx?.claimsFor(state)
if (newClaims?.size) {
// The log is the source of truth: (re)assert claims for the caught-up state.
const missing = [...newClaims.keys()].filter((k) => !oldClaims!.has(k))
if (missing.length) await this.#ctx.driver.setMany(missing.map((k) => [k, [t.esc, this.#ctx.now()] satisfies Claim]))
}
const releases = oldClaims ? [...oldClaims.keys()].filter((k) => !newClaims!.has(k)) : []
const projection: Projection = state ? { state, version, commit } : { deleted: true, version, commit }
await this.#writeProjection(t, cur, state, projection, releases)
return true
}
async *#projectionScan(): AsyncGenerator<Hit> {
const { prefix, end } = this.#childRange()
for await (const hit of scanChildren(this.#ctx.driver, prefix, end, this.#ctx.sep, { values: true })) {
const p = hit.value as Projection | undefined
if (hit.hasOwn && p?.state) yield { esc: hit.esc, state: p.state }
}
}
async *#indexScan(plan: Plan, want: number): AsyncGenerator<Hit> {
const idx = this.#childIdx!
const { driver, readRepair } = this.#ctx
const parentProj = this.#ctx.projBase + this.#prefix
const seen = new Set<string>()
let chunk: string[] = []
let chunkSize = Math.min(FIND_CHUNK, Math.max(1, want))
const it = driver.scan({ gte: plan.gte, lt: plan.lt, reverse: plan.reverse })[Symbol.asyncIterator]()
async function* flush(): AsyncGenerator<Hit> {
const entries = chunk
chunk = []
chunkSize = FIND_CHUNK
const escs = entries.map((e) => idx.childOf(plan.index, e))
const projs = (await driver.getMany(escs.map((e) => parentProj + e))) as (Projection | undefined)[]
const stale: string[] = []
try {
for (let i = 0; i < entries.length; i++) {
const esc = escs[i]!
const state = projs[i]?.state
if (!idx.entriesFor(esc, state).has(entries[i]!)) {
stale.push(entries[i]!)
continue
}
if (seen.has(esc)) continue
seen.add(esc)
yield { esc, state: state! }
}
} finally {
if (readRepair && stale.length) await driver.deleteMany(stale)
}
}
try {
for (;;) {
const r = await it.next()
if (r.done) break
chunk.push(r.value[0])
if (chunk.length >= chunkSize) yield* flush()
}
if (chunk.length) yield* flush()
} finally {
await it.return?.()
}
}
#warnFullScan(filter: Filter): void {
if (!this.#ctx.warnOnFullScan) return
const shape = JSON.stringify(filterShape(filter))
if (this.#ctx.warned.has(shape)) return
this.#ctx.warned.add(shape)
console.warn(`[potential-storage] find() on "${this.selfKey || '(root)'}" has no usable index; full scan for filter ${shape}`)
}
}
/** Filter with values replaced by `?`, for de-duplicating warnings. */
function filterShape(f: unknown): unknown {
if (Array.isArray(f)) return f.map(filterShape)
if (isPlainObject(f))
return Object.fromEntries(
Object.entries(f).map(([k, v]) => [
k,
k.startsWith('$') && (k === '$and' || k === '$or' || k === '$not') ? filterShape(v) : isPlainObject(v) ? filterShape(v) : '?',
]),
)
return '?'
}
function comparator(sort: [string, 1 | -1][]): (a: Hit, b: Hit) => number {
const tieDir = sort[0]?.[1] ?? 1
return (a, b) => {
for (const [field, dir] of sort) {
const d = compareValues(sortKey(a.state, field, dir), sortKey(b.state, field, dir))
if (d !== 0) return d * dir
}
return a.esc === b.esc ? 0 : (a.esc < b.esc ? -1 : 1) * tieDir
}
}
function projector(spec: Record<string, 0 | 1 | boolean>): (d: Doc) => Doc {
const fields = Object.entries(spec)
const include = fields.some(([, v]) => v === 1 || v === true)
if (include && fields.some(([, v]) => v === 0 || v === false)) throw new TypeError('Projection cannot mix inclusion and exclusion')
return (d) => {
if (include) {
const out: Doc = {}
for (const [f] of fields) {
const v = getPath(d, f)
if (v !== undefined) setPath(out, f, clone(v))
}
return out
}
const out = clone(d)
for (const [f] of fields) unsetPath(out, f)
return out
}
}
/** Event-sourced document storage: the root document node. */
export class EventSourcedDocumentStorage<S extends Spec = AnySpec> extends EventSourcedDocumentNodeImpl<S> {
constructor(events: EventNode<AnySpec>, opts: EventSourcedDocumentStorageOptions = {}) {
let driver: KVDriver
let sep: string
let projBase: string
if (opts.projections) {
driver = opts.projections.driver
sep = opts.projections.separator
projBase = opts.projections.prefix
} else if (events instanceof KVEventNode) {
const kv = events.kv
driver = kv.driver
sep = kv.separator
projBase = kv.prefix + RESERVED.projections + sep
} else {
throw new TypeError('EventSourcedDocumentStorage needs `projections` when the event store is not KV-backed')
}
const registry = new DocIndexRegistry(opts.indexes ?? {}, sep)
if (registry.hasUnique && !driver.setIfAbsent) throw new CapabilityError('setIfAbsent', 'unique document indexes require it')
const ctx: DocContext = {
driver,
sep,
projBase,
registry,
readRepair: opts.readRepair ?? true,
maxRetries: opts.maxRetries ?? 5,
warnOnFullScan: opts.warnOnFullScan ?? (typeof process === 'undefined' || process.env?.NODE_ENV !== 'production'),
warned: new Set(),
newId: opts.newId ?? ulid,
claimGraceMs: opts.claimGraceMs ?? 30_000,
now: opts.now ?? Date.now,
writeLock: new KeyedMutex(),
}
super(ctx, events, undefined, [], '', '', registry.trie.rootState, undefined)
}
}
src/documents/filter.ts
-155
View File
@@ -1,155 +0,0 @@
import type { IndexValue } from '../core/encoding.js'
import { compareValues, deepEqual, isPlainObject, isScalar, resolvePath, typeRank, type Doc } from './values.js'
/**
* Query filter (Mongo subset): field conditions with implicit equality and dot paths,
* `$eq $ne $gt $gte $lt $lte $in $nin $exists`, and `$and $or $not`.
*
* Semantics differences from MongoDB (see DESIGN.md §12, M6):
* - Comparisons are type-bracketed: `$gt: 5` only matches numbers.
* - On array fields, all positive operators in one condition (`$eq $gt $gte $lt $lte $in`) must be
* satisfied by the same element (like `$elemMatch`), so index range scans are exact.
*/
export type Filter = { [field: string]: unknown; $and?: Filter[]; $or?: Filter[]; $not?: Filter }
const POSITIVE = new Set(['$eq', '$gt', '$gte', '$lt', '$lte', '$in'])
const FIELD_LEVEL = new Set(['$ne', '$nin', '$exists', '$not'])
export function isOperatorObject(v: unknown): v is Record<string, unknown> {
if (!isPlainObject(v)) return false
const keys = Object.keys(v)
return keys.length > 0 && keys.every((k) => k.startsWith('$'))
}
export function matches(doc: unknown, filter: Filter): boolean {
for (const key of Object.keys(filter)) {
const cond = filter[key]
switch (key) {
case '$and':
if (!(cond as Filter[]).every((f) => matches(doc, f))) return false
break
case '$or':
if (!(cond as Filter[]).some((f) => matches(doc, f))) return false
break
case '$not':
if (matches(doc, cond as Filter)) return false
break
default:
if (key.startsWith('$')) throw new TypeError(`Unknown top-level operator ${key}`)
if (!matchesField(resolvePath(doc, key), cond)) return false
}
}
return true
}
/** Whether the values found at a path satisfy a field condition. */
export function matchesField(values: unknown[], cond: unknown): boolean {
if (!isOperatorObject(cond)) return someCandidate(values, (c) => eq(c, cond))
const positive: [string, unknown][] = []
for (const [op, arg] of Object.entries(cond)) {
if (POSITIVE.has(op)) positive.push([op, arg])
else if (!FIELD_LEVEL.has(op)) throw new TypeError(`Unknown operator ${op}`)
}
for (const [op, arg] of Object.entries(cond)) {
switch (op) {
case '$ne':
if (someCandidate(values, (c) => eq(c, arg))) return false
break
case '$nin':
if (someCandidate(values, (c) => (arg as unknown[]).some((x) => eq(c, x)))) return false
break
case '$exists':
if (values.length > 0 !== Boolean(arg)) return false
break
case '$not':
if (matchesField(values, arg)) return false
break
}
}
if (positive.length === 0) return true
return someCandidate(values, (c) => positive.every(([op, arg]) => applyPositive(op, c, arg)))
}
/** Tests each value and, for arrays, each element. Missing paths test `undefined` once. */
function someCandidate(values: unknown[], test: (c: unknown) => boolean): boolean {
if (values.length === 0) return test(undefined)
for (const v of values) {
if (test(v)) return true
if (Array.isArray(v) && v.some(test)) return true
}
return false
}
function eq(c: unknown, x: unknown): boolean {
if (x === null) return c === null || c === undefined
return deepEqual(c, x)
}
function applyPositive(op: string, c: unknown, arg: unknown): boolean {
switch (op) {
case '$eq':
return eq(c, arg)
case '$in':
if (!Array.isArray(arg)) throw new TypeError('$in expects an array')
return arg.some((x) => eq(c, x))
default: {
const rc = typeRank(c)
if (rc !== typeRank(arg) || rc === 4) return false
const d = compareValues(c, arg)
return op === '$gt' ? d > 0 : op === '$gte' ? d >= 0 : op === '$lt' ? d < 0 : d <= 0
}
}
}
// ------------------------------------------------------------------ planner analysis
export interface RangePred {
gt?: IndexValue
gte?: IndexValue
lt?: IndexValue
lte?: IndexValue
}
export interface FieldPred {
eq?: IndexValue
range?: RangePred
}
/**
* Index-usable predicates per field, from the top level and top-level `$and`.
* Only scalar `$eq`/implicit equality and same-typed scalar range bounds qualify.
*/
export function analyze(filter: Filter, out = new Map<string, FieldPred>()): Map<string, FieldPred> {
for (const key of Object.keys(filter)) {
const cond = filter[key]
if (key === '$and') {
for (const f of cond as Filter[]) analyze(f, out)
continue
}
if (key.startsWith('$')) continue
const pred = out.get(key) ?? {}
if (!isOperatorObject(cond)) {
if (isScalar(cond)) pred.eq ??= cond
} else {
if ('$eq' in cond && isScalar(cond.$eq)) pred.eq ??= cond.$eq
const range: RangePred = {}
let rank = -1
let ok = false
for (const op of ['$gt', '$gte', '$lt', '$lte'] as const) {
if (!(op in cond)) continue
const v = cond[op]
if (!isScalar(v)) continue
const r = typeRank(v)
if (rank !== -1 && r !== rank) continue
rank = r
range[op.slice(1) as keyof RangePred] = v
ok = true
}
if (ok && !pred.range) pred.range = range
}
if (pred.eq !== undefined || pred.range) out.set(key, pred)
}
return out
}
export type { Doc }
src/documents/index.ts
-7
View File
@@ -1,7 +0,0 @@
export * from './types.js'
export * from './event-sourced.js'
export * from './filter.js'
export * from './update.js'
export * from './indexes.js'
export * from './ulid.js'
export { compareValues, deepEqual, type Doc } from './values.js'
src/documents/indexes.ts
-220
View File
@@ -1,220 +0,0 @@
import { encodeIndexValues, indexValueCodec, type IndexValue } from '../core/encoding.js'
import { escapeSegment, RESERVED, succ } from '../core/path.js'
import { PatternTrie, patternsOverlap, type TrieState } from '../core/trie.js'
import type { FieldPred, RangePred } from './filter.js'
import { indexScalars, resolvePath, typeRank, type Doc } from './values.js'
export interface DocIndexSpec {
/** Default: the fields joined with `,`. */
name?: string
fields: string[]
/** Claims `~u/<name>/<values>` per value tuple. Documents where every field is missing/null are not claimed. */
unique?: boolean
}
/** Path pattern (`*` = one segment, `''` = root) → indexes over that node's direct children. */
export type DocIndexDeclarations = Record<string, DocIndexSpec[]>
export interface SortSpec {
[field: string]: 1 | -1
}
interface CompiledDocIndex {
name: string
fields: string[]
segs: string[][]
unique: boolean
/** `escapeSegment(name) + sep` */
head: string
}
export interface Plan {
index: string
gte: string
lt: string
reverse: boolean
/** Index order satisfies the requested sort. */
sorted: boolean
}
/** Document indexes over the children of one node. */
export class DocIndexes {
readonly #list: CompiledDocIndex[]
readonly #byName: Map<string, CompiledDocIndex>
readonly #enc: (v: IndexValue) => string
constructor(
list: CompiledDocIndex[],
/** `<projBase><nodePrefix>~i<sep>` */
readonly base: string,
/** `<projBase><nodePrefix>~u<sep>` */
readonly uniqueBase: string,
readonly sep: string,
) {
this.#list = list
this.#byName = new Map(list.map((i) => [i.name, i]))
const codec = indexValueCodec(sep)
this.#enc = (v) => codec.encode(v)
}
get hasUnique(): boolean {
return this.#list.some((i) => i.unique)
}
/** Value tuples a state produces for one index (cartesian over multikey fields). */
#tuples(idx: CompiledDocIndex, state: Doc): IndexValue[][] {
let tuples: IndexValue[][] = [[]]
let multi = 0
for (const segs of idx.segs) {
const found = resolvePath(state, segs)
const scalars =
found.length === 0 ? [null] : found.length === 1 && !Array.isArray(found[0]) ? indexScalars(found[0]) : [...new Set(found.flatMap(indexScalars))]
if (tuples.length === 1 && scalars.length === 1) {
tuples[0]!.push(scalars[0]!)
continue
}
if (scalars.length > 1 && ++multi > 1) throw new TypeError(`Index "${idx.name}": cannot index more than one array field per document`)
tuples = tuples.flatMap((t) => scalars.map((s) => [...t, s]))
}
return tuples
}
entriesFor(escChild: string, state: Doc | undefined, out: Set<string> = new Set()): Set<string> {
if (!state) return out
for (const idx of this.#list) {
const stem = this.base + idx.head
for (const t of this.#tuples(idx, state)) out.add(stem + encodeIndexValues(t, this.sep) + this.sep + escChild)
}
return out
}
/** Unique claim key → index name and values. */
claimsFor(state: Doc | undefined): Map<string, { index: string; values: IndexValue[] }> {
const out = new Map<string, { index: string; values: IndexValue[] }>()
if (!state) return out
for (const idx of this.#list) {
if (!idx.unique) continue
for (const t of this.#tuples(idx, state)) {
if (t.every((v) => v === null)) continue
out.set(this.uniqueBase + idx.head + encodeIndexValues(t, this.sep), { index: idx.name, values: t })
}
}
return out
}
/** Escaped child segment of an entry key of index `name`. */
childOf(name: string, entryKey: string): string {
const idx = this.#byName.get(name)!
let pos = this.base.length + idx.head.length
for (let i = 0; i < idx.fields.length; i++) pos = entryKey.indexOf(this.sep, pos) + 1
return entryKey.slice(pos)
}
/**
* Chooses the index covering the most leading equality fields, then at most one range field.
* Ties prefer an index whose order satisfies `sort`. With no usable predicate, an index whose
* order satisfies `sort` is still used (a full, ordered index scan). Returns `undefined` for a
* full projection scan.
*/
plan(preds: Map<string, FieldPred>, sort: [string, 1 | -1][]): Plan | undefined {
let best: { idx: CompiledDocIndex; k: number; range: RangePred | undefined; sorted: boolean; dir: 1 | -1; score: number } | undefined
for (const idx of this.#list) {
let k = 0
while (k < idx.fields.length && preds.get(idx.fields[k]!)?.eq !== undefined) k++
const range = k < idx.fields.length ? preds.get(idx.fields[k]!)?.range : undefined
// Sort fields constrained by equality are constant; drop them before matching.
const eqFields = new Set(idx.fields.slice(0, k))
const rest = sort.filter(([f]) => !eqFields.has(f))
// Ties are broken by child key in the direction of the first sort field.
const dir = rest[0]?.[1] ?? sort[0]?.[1] ?? 1
const sorted = rest.length === 0 || (rest.every(([f, d], i) => idx.fields[k + i] === f && d === dir) && rest.length <= idx.fields.length - k)
const score = k * 2 + (range ? 1 : 0)
if (score === 0 && !(sorted && rest.length > 0)) continue
if (!best || score > best.score || (score === best.score && sorted && !best.sorted)) best = { idx, k, range, sorted, dir, score }
}
if (!best) return undefined
const { idx, k, range, sorted, dir } = best
const succSep = succ(this.sep)
let prefix = this.base + idx.head
for (let i = 0; i < k; i++) prefix += this.#enc(preds.get(idx.fields[i]!)!.eq!) + this.sep
// prefix ends with sep: the range of everything under it is [prefix, prefix-without-sep + succ(sep)).
const all = { gte: prefix, lt: prefix.slice(0, -1) + succSep }
let { gte, lt } = all
if (range) {
const sample = (range.gt ?? range.gte ?? range.lt ?? range.lte) as IndexValue
const [lo, hi] = typeBounds(typeRank(sample), this.#enc)
gte = prefix + lo
lt = hi === undefined ? all.lt : prefix + hi
if (range.gte !== undefined) gte = max(gte, prefix + this.#enc(range.gte))
if (range.gt !== undefined) gte = max(gte, prefix + this.#enc(range.gt) + succSep)
if (range.lt !== undefined) lt = min(lt, prefix + this.#enc(range.lt))
if (range.lte !== undefined) lt = min(lt, prefix + this.#enc(range.lte) + succSep)
}
return { index: idx.name, gte, lt, reverse: sorted && dir === -1, sorted }
}
}
/** Encoded `[lower, upper)` of one type class; `undefined` upper = end of index. */
function typeBounds(rank: number, enc: (v: IndexValue) => string): [string, string | undefined] {
switch (rank) {
case 0:
return [enc(null), enc(false)]
case 1:
return [enc(false), enc(-Infinity)]
case 2:
return [enc(-Infinity), enc('')]
default:
return [enc(''), undefined]
}
}
const max = (a: string, b: string) => (a > b ? a : b)
const min = (a: string, b: string) => (a < b ? a : b)
export class DocIndexRegistry {
readonly trie: PatternTrie<CompiledDocIndex[]>
readonly empty: boolean
readonly hasUnique: boolean
constructor(
decls: DocIndexDeclarations,
readonly sep: string,
) {
this.trie = new PatternTrie(sep)
const patterns = Object.keys(decls)
const compiled = new Map<string, CompiledDocIndex[]>()
for (const p of patterns) {
const list = decls[p]!.map((spec): CompiledDocIndex => {
if (!spec.fields.length) throw new TypeError(`Index on "${p}" has no fields`)
const name = spec.name ?? spec.fields.join(',')
return { name, fields: spec.fields, segs: spec.fields.map((f) => f.split('.')), unique: spec.unique === true, head: escapeSegment(name, sep) + sep }
})
const names = list.map((i) => i.name)
const dup = names.find((n, i) => names.indexOf(n) !== i)
if (dup) throw new TypeError(`Duplicate index name "${dup}" on "${p}"`)
compiled.set(p, list)
}
for (let i = 0; i < patterns.length; i++) {
for (let j = i + 1; j < patterns.length; j++) {
const a = patterns[i]!
const b = patterns[j]!
if (!patternsOverlap(a, b, sep)) continue
const bn = new Set(compiled.get(b)!.map((x) => x.name))
const clash = compiled.get(a)!.filter((x) => bn.has(x.name))
if (clash.length) throw new TypeError(`Index name(s) ${clash.map((c) => c.name).join(', ')} declared on overlapping patterns "${a}" and "${b}"`)
}
}
for (const p of patterns) this.trie.add(p, compiled.get(p)!)
this.empty = patterns.length === 0
this.hasUnique = [...compiled.values()].some((l) => l.some((i) => i.unique))
}
/** `projPrefix` is the projection-space prefix of the node (`<projBase><nodePrefix>`). */
bind(state: TrieState<CompiledDocIndex[]>, projPrefix: string): DocIndexes | undefined {
if (this.empty) return undefined
const found = PatternTrie.values(state)
if (found.length === 0) return undefined
const list = found.flatMap((f) => f.value)
return new DocIndexes(list, projPrefix + RESERVED.index + this.sep, projPrefix + RESERVED.unique + this.sep, this.sep)
}
}
src/documents/types.ts
-90
View File
@@ -1,90 +0,0 @@
import type { StoredEvent } from '../events/types.js'
import type { AnySpec, ChildOf, KeyOf, KeyString, Spec, ValueOf } from '../types/schema.js'
import type { Filter } from './filter.js'
import type { SortSpec } from './indexes.js'
import type { DocEvent, UpdateSpec } from './update.js'
export interface FindOptions {
sort?: SortSpec
limit?: number
skip?: number
/** Inclusion (`{ a: 1 }`) or exclusion (`{ a: 0 }`) of dot-path fields. */
projection?: Record<string, 0 | 1 | boolean>
}
export interface FindResult<T> {
key: string
data: T
}
export interface GetOptions {
/** Check the stream head and catch the projection up first (one extra scan). */
consistent?: boolean
}
export interface UpdateOptions {
/** Treat a missing document as `{}`. */
upsert?: boolean
}
export interface DocKeysOptions {
limit?: number
after?: string | number
reverse?: boolean
}
/** Document type of a node's children. */
export type ChildDoc<S> = ValueOf<ChildOf<S, KeyOf<S>>>
/**
* One node of a document store: a document (its own) and a collection of child documents.
* Same navigation and self/child arity rules as `KVNode`; in forms with an optional leading key,
* a non-null object first argument selects the self form.
*/
export interface DocumentNode<S extends Spec = AnySpec> {
readonly path: readonly string[]
child<K extends KeyOf<S>>(key: K): DocumentNode<ChildOf<S, K>>
as<S2 extends Spec>(): DocumentNode<S2>
get(opts?: GetOptions): Promise<ValueOf<S> | undefined>
get<K extends KeyOf<S>>(key: K, opts?: GetOptions): Promise<ValueOf<ChildOf<S, K>> | undefined>
set(doc: ValueOf<S>): Promise<void>
set<K extends KeyOf<S>>(key: K, doc: ValueOf<ChildOf<S, K>>): Promise<void>
/** Inserts a child with a generated time-sortable id; returns the id. */
insert(doc: ChildDoc<S>): Promise<string>
/** Returns the new state, or `undefined` if the document doesn't exist (and no `upsert`). */
update(spec: UpdateSpec, opts?: UpdateOptions): Promise<ValueOf<S> | undefined>
update<K extends KeyOf<S>>(key: K, spec: UpdateSpec, opts?: UpdateOptions): Promise<ValueOf<ChildOf<S, K>> | undefined>
/** Returns whether a document was deleted. */
delete(): Promise<boolean>
delete<K extends KeyOf<S>>(key: K): Promise<boolean>
/** Queries direct children. */
find(filter?: Filter, opts?: FindOptions): AsyncIterable<FindResult<ChildDoc<S>>>
findOne(filter?: Filter, opts?: FindOptions): Promise<FindResult<ChildDoc<S>> | undefined>
/** Keys of direct children that hold a (non-deleted) document. */
keys(opts?: DocKeysOptions): AsyncIterable<KeyString<S>>
}
/** Document store backed by an event log: adds history and point-in-time reads. */
export interface EventSourcedDocumentNode<S extends Spec = AnySpec> extends DocumentNode<S> {
child<K extends KeyOf<S>>(key: K): EventSourcedDocumentNode<ChildOf<S, K>>
as<S2 extends Spec>(): EventSourcedDocumentNode<S2>
history(): AsyncIterable<StoredEvent<DocEvent>>
history<K extends KeyOf<S>>(key: K): AsyncIterable<StoredEvent<DocEvent>>
/** State as of `version` (`undefined` if missing or deleted then). */
getAt(version: number): Promise<ValueOf<S> | undefined>
getAt<K extends KeyOf<S>>(key: K, version: number): Promise<ValueOf<ChildOf<S, K>> | undefined>
/** Rebuilds lagging projections for this node and all descendants; returns how many were repaired. */
catchUp(): Promise<number>
}
export type DocumentStorage<S extends Spec = AnySpec> = DocumentNode<S>
src/documents/ulid.ts
-37
View File
@@ -1,37 +0,0 @@
const ALPHABET = '0123456789ABCDEFGHJKMNPQRSTVWXYZ' // Crockford base32
/**
* ULID-like id generator: 10 chars of millisecond timestamp + 16 chars of randomness,
* lexicographically sortable by creation time. Ids generated within the same millisecond
* increment the random part, so they stay strictly increasing per generator.
*/
export function createIdGenerator(now: () => number = Date.now): () => string {
let lastTime = -1
const rand = new Uint8Array(16) // one base32 digit (0–31) per byte
return () => {
let t = now()
if (t <= lastTime) {
t = lastTime
// Increment the random part (base 32, right to left).
let i = 15
while (i >= 0 && rand[i] === 31) rand[i--] = 0
if (i < 0) throw new Error('ULID random overflow within one millisecond')
rand[i]!++
} else {
crypto.getRandomValues(rand)
for (let i = 0; i < 16; i++) rand[i]! &= 31
// Leave headroom so increments rarely overflow.
rand[0]! &= 15
lastTime = t
}
let out = ''
for (let i = 9; i >= 0; i--) {
out = ALPHABET[t % 32]! + out
t = Math.floor(t / 32)
}
for (let i = 0; i < 16; i++) out += ALPHABET[rand[i]!]
return out
}
}
export const ulid = createIdGenerator()
src/documents/update.ts
-109
View File
@@ -1,109 +0,0 @@
import { isOperatorObject, matches, matchesField, type Filter } from './filter.js'
import { clone, deepEqual, getPath, isPlainObject, setPath, unsetPath, type Doc } from './values.js'
/** Update operators supported in v1. Paths may use dots. */
export interface UpdateSpec {
$set?: Record<string, unknown>
$unset?: Record<string, unknown>
$inc?: Record<string, number>
$push?: Record<string, unknown>
$pull?: Record<string, unknown>
$addToSet?: Record<string, unknown>
}
/** Stored document events: resolved deltas, never operators, so replay is deterministic. */
export type DocEvent = { op: 'set'; doc: Doc } | { op: 'patch'; set?: Record<string, unknown>; unset?: string[] } | { op: 'delete' }
export interface Patch {
set: Record<string, unknown>
unset: string[]
}
const OPS = new Set(['$set', '$unset', '$inc', '$push', '$pull', '$addToSet'])
/** Resolves operators against `state` into a patch of final values. */
export function resolveUpdate(state: Doc, spec: UpdateSpec): Patch {
const keys = Object.keys(spec)
if (keys.length === 0) throw new TypeError('Empty update')
for (const k of keys) if (!OPS.has(k)) throw new TypeError(`Unsupported update operator ${k}`)
const patch: Patch = { set: {}, unset: [] }
const touched: string[] = []
const touch = (path: string) => {
for (const t of touched) {
if (t === path || t.startsWith(path + '.') || path.startsWith(t + '.')) throw new TypeError(`Update paths conflict: "${t}" and "${path}"`)
}
touched.push(path)
}
const arrayAt = (path: string, op: string): unknown[] => {
const cur = getPath(state, path)
if (cur === undefined || cur === null) return []
if (!Array.isArray(cur)) throw new TypeError(`${op} on non-array field "${path}"`)
return cur
}
const eachOf = (v: unknown): unknown[] => (isPlainObject(v) && Object.keys(v).length === 1 && Array.isArray(v.$each) ? (v.$each as unknown[]) : [v])
for (const [path, v] of Object.entries(spec.$set ?? {})) {
touch(path)
patch.set[path] = clone(v)
}
for (const path of Object.keys(spec.$unset ?? {})) {
touch(path)
patch.unset.push(path)
}
for (const [path, by] of Object.entries(spec.$inc ?? {})) {
touch(path)
if (typeof by !== 'number') throw new TypeError(`$inc by non-number at "${path}"`)
const cur = getPath(state, path)
if (cur !== undefined && cur !== null && typeof cur !== 'number') throw new TypeError(`$inc on non-number field "${path}"`)
patch.set[path] = ((cur as number | undefined) ?? 0) + by
}
for (const [path, v] of Object.entries(spec.$push ?? {})) {
touch(path)
patch.set[path] = [...clone(arrayAt(path, '$push')), ...clone(eachOf(v))]
}
for (const [path, v] of Object.entries(spec.$addToSet ?? {})) {
touch(path)
const next = clone(arrayAt(path, '$addToSet'))
for (const item of eachOf(v)) if (!next.some((x) => deepEqual(x, item))) next.push(clone(item))
patch.set[path] = next
}
for (const [path, cond] of Object.entries(spec.$pull ?? {})) {
touch(path)
patch.set[path] = clone(arrayAt(path, '$pull')).filter((el) => !pullMatches(el, cond))
}
return patch
}
function pullMatches(el: unknown, cond: unknown): boolean {
if (isOperatorObject(cond)) return matchesField([el], cond)
if (isPlainObject(cond) && isPlainObject(el)) return matches(el, cond as Filter)
return deepEqual(el, cond)
}
/** Applies a resolved patch to a copy of `state`. */
export function applyPatch(state: Doc, patch: { set?: Record<string, unknown>; unset?: string[] }): Doc {
const next = clone(state)
for (const [path, v] of Object.entries(patch.set ?? {})) setPath(next, path, clone(v))
for (const path of patch.unset ?? []) unsetPath(next, path)
return next
}
/** Folds one event into a state (`undefined` = no document). */
export function applyEvent(state: Doc | undefined, e: DocEvent): Doc | undefined {
switch (e.op) {
case 'set':
return clone(e.doc)
case 'patch':
return applyPatch(state ?? {}, e)
case 'delete':
return undefined
}
}
export function toPatchEvent(p: Patch): DocEvent {
const e: { op: 'patch'; set?: Record<string, unknown>; unset?: string[] } = { op: 'patch' }
if (Object.keys(p.set).length) e.set = p.set
if (p.unset.length) e.unset = p.unset
return e
}
src/documents/values.ts
-163
View File
@@ -1,163 +0,0 @@
import type { IndexValue } from '../core/encoding.js'
/** Plain JSON object. */
export type Doc = Record<string, unknown>
export function isPlainObject(v: unknown): v is Doc {
return typeof v === 'object' && v !== null && !Array.isArray(v)
}
export function isScalar(v: unknown): v is IndexValue {
return v === null || typeof v === 'string' || typeof v === 'boolean' || (typeof v === 'number' && !Number.isNaN(v))
}
export function splitPath(path: string): string[] {
return path.split('.')
}
/**
* Values reachable at a dot path. Arrays are traversed: a non-numeric segment applied to an
* array maps over its elements. Missing paths yield no values.
*/
export function resolvePath(doc: unknown, path: string | readonly string[]): unknown[] {
const segs = typeof path === 'string' ? splitPath(path) : path
const out: unknown[] = []
walk(doc, segs, 0, out)
return out
}
function walk(v: unknown, segs: readonly string[], i: number, out: unknown[]): void {
if (i === segs.length) {
if (v !== undefined) out.push(v)
return
}
const seg = segs[i]!
if (Array.isArray(v)) {
if (/^\d+$/.test(seg)) walk(v[Number(seg)], segs, i + 1, out)
else for (const el of v) if (isPlainObject(el) || Array.isArray(el)) walk(el, segs, i, out)
return
}
if (isPlainObject(v) && Object.hasOwn(v, seg)) walk(v[seg], segs, i + 1, out)
}
/** Value at a dot path without array traversal (numeric segments index arrays). */
export function getPath(doc: unknown, path: string): unknown {
let v = doc
for (const seg of splitPath(path)) {
if (Array.isArray(v) && /^\d+$/.test(seg)) v = v[Number(seg)]
else if (isPlainObject(v) && Object.hasOwn(v, seg)) v = v[seg]
else return undefined
}
return v
}
/** Sets a dot path in place, creating intermediate objects. */
export function setPath(doc: Doc, path: string, value: unknown): void {
const segs = splitPath(path)
let v: unknown = doc
for (let i = 0; i < segs.length - 1; i++) {
const seg = segs[i]!
const container = v as Record<string, unknown>
let next = Array.isArray(container) && /^\d+$/.test(seg) ? container[Number(seg)] : container[seg]
if (next === undefined || next === null) {
next = {}
if (Array.isArray(container)) container[Number(seg)] = next
else container[seg] = next
} else if (typeof next !== 'object') {
throw new TypeError(`Cannot set "${path}": "${segs.slice(0, i + 1).join('.')}" is not an object`)
}
v = next
}
const last = segs.at(-1)!
if (Array.isArray(v) && /^\d+$/.test(last)) v[Number(last)] = value
else (v as Doc)[last] = value
}
export function unsetPath(doc: Doc, path: string): void {
const segs = splitPath(path)
let v: unknown = doc
for (let i = 0; i < segs.length - 1; i++) {
v = isPlainObject(v) ? v[segs[i]!] : Array.isArray(v) && /^\d+$/.test(segs[i]!) ? v[Number(segs[i])] : undefined
if (typeof v !== 'object' || v === null) return
}
const last = segs.at(-1)!
if (Array.isArray(v) && /^\d+$/.test(last))
v[Number(last)] = null // Mongo semantics: keep positions
else if (isPlainObject(v)) delete v[last]
}
export function deepEqual(a: unknown, b: unknown): boolean {
if (a === b) return true
if (typeof a !== 'object' || typeof b !== 'object' || a === null || b === null) return false
if (Array.isArray(a) !== Array.isArray(b)) return false
if (Array.isArray(a)) {
const bb = b as unknown[]
return a.length === bb.length && a.every((x, i) => deepEqual(x, bb[i]))
}
const ka = Object.keys(a)
const kb = Object.keys(b)
return ka.length === kb.length && ka.every((k) => Object.hasOwn(b, k) && deepEqual((a as Doc)[k], (b as Doc)[k]))
}
/** Deep copy of a JSON-compatible value (documents are JSON). */
export function clone<T>(v: T): T {
if (v === null || typeof v !== 'object') return v
if (Array.isArray(v)) return v.map(clone) as T
const out: Doc = {}
for (const k in v) out[k] = clone((v as Doc)[k])
return out as T
}
/** Type class used for bracketing comparisons, in index-encoding order. */
export function typeRank(v: unknown): number {
if (v === null || v === undefined) return 0
if (typeof v === 'boolean') return 1
if (typeof v === 'number') return 2
if (typeof v === 'string') return 3
return 4
}
/**
* Total order matching the index encoding for scalars:
* `null/missing < false < true < numbers < strings < anything else`.
*/
export function compareValues(a: unknown, b: unknown): number {
const ra = typeRank(a)
const rb = typeRank(b)
if (ra !== rb) return ra < rb ? -1 : 1
if (ra === 1 || ra === 2 || ra === 3) return a === b ? 0 : (a as number) < (b as number) ? -1 : 1
if (ra === 4) {
const sa = JSON.stringify(a)
const sb = JSON.stringify(b)
return sa === sb ? 0 : sa < sb ? -1 : 1
}
return 0
}
/**
* The scalar(s) an index stores for a field value: scalars as-is, arrays as their scalar
* elements (multikey; empty → null), and missing or non-scalar values as `null`.
*/
export function indexScalars(v: unknown): IndexValue[] {
if (Array.isArray(v)) {
const out: IndexValue[] = []
for (const el of v) out.push(isScalar(el) ? el : null)
return out.length ? out : [null]
}
return [isScalar(v) ? v : null]
}
/** Sort key for a field under a direction; arrays sort by their min (asc) or max (desc) element. */
export function sortKey(doc: unknown, path: string, dir: 1 | -1): unknown {
const vals = resolvePath(doc, path)
if (vals.length === 0) return null
let best: unknown
let first = true
for (const v of vals) {
for (const s of indexScalars(v)) {
if (first || compareValues(s, best) * dir < 0) best = s
first = false
}
}
return best
}
src/errors.ts
-50
View File
@@ -1,50 +0,0 @@
/** Base class for all PotentialStorage errors. */
export class PotentialStorageError extends Error {
constructor(message: string) {
super(message)
this.name = new.target.name
}
}
/** An optimistic-concurrency check failed (stale `expectedVersion` or a lost commit race). */
export class ConcurrencyError extends PotentialStorageError {
constructor(
readonly stream: string,
readonly expectedVersion: number | 'any',
readonly actualVersion: number,
) {
super(`Concurrency conflict on "${stream}": expected version ${expectedVersion}, actual ${actualVersion}`)
}
}
/** A unique index already holds the given value for another key. */
export class UniqueConstraintError extends PotentialStorageError {
constructor(
readonly index: string,
readonly values: readonly unknown[],
readonly existingKey?: string,
) {
super(`Unique index "${index}" already contains ${JSON.stringify(values)}${existingKey === undefined ? '' : ` (key "${existingKey}")`}`)
}
}
/** The driver lacks an optional capability required by the requested feature. */
export class CapabilityError extends PotentialStorageError {
constructor(
readonly capability: string,
detail?: string,
) {
super(`Driver lacks capability "${capability}"${detail ? `: ${detail}` : ''}`)
}
}
/**
* A key argument is invalid: passed but `undefined`/`null`, empty, or otherwise unusable.
* Extends `TypeError` so callers can catch either.
*/
export class InvalidKeyError extends TypeError {
constructor(message: string) {
super(message)
this.name = 'InvalidKeyError'
}
}
src/event-storage/base-event-storage.ts
+100
View File
@@ -0,0 +1,100 @@
import { Keys } from '../utils/keys.js'
import type { AnySpec, ChildOf, KeyOf, KeyString, Spec, ValueAt, ValueOf } from '../utils/schema.js'
export interface AppendOptions {
/** Version the stream must be at (0 = empty). Default `'any'`. */
expectedVersion?: number | 'any'
}
export interface AppendResult {
/** Stream version after the append. */
version: number
}
export interface ReadOptions {
/** First version to read (inclusive). Default 1. */
from?: number
/** Last version to read (inclusive). Default: the head. */
to?: number
limit?: number
}
export interface StoredEvent<E> {
version: number
/** Append timestamp (ms since epoch). */
at: number
data: E
}
export interface StreamKeysOptions {
limit?: number
after?: string | number
reverse?: boolean
}
/**
* Append-only event streams. Every node is a stream (its own events) and a parent of child
* streams. Implementations provide four primitives over stream paths; the user-facing,
* self-or-child API is defined here once.
*/
export abstract class BaseEventStorage<S extends Spec = AnySpec> {
/** Unescaped path of this node (`[]` for the root, which is not a stream). */
path: readonly string[] = []
abstract appendTo(stream: readonly string[], events: unknown[], expectedVersion: number | 'any'): Promise<AppendResult>
abstract readFrom(stream: readonly string[], options: ReadOptions): AsyncIterable<StoredEvent<unknown>>
abstract versionOf(stream: readonly string[]): Promise<number>
abstract streamsOf(parent: readonly string[], options: StreamKeysOptions): AsyncIterable<string>
child<K extends KeyOf<S>>(key: K): BaseEventStorage<ChildOf<S, K>> {
const node = Object.create(this) as BaseEventStorage<ChildOf<S, K>>
node.path = [...this.path, Keys.segment(key)]
return node
}
as<S2 extends Spec>(): BaseEventStorage<S2> {
return this as unknown as BaseEventStorage<S2>
}
/** `append(events, options?)` on this stream; `append(key, events, options?)` on a child stream. */
append<K extends KeyOf<S>>(
...args: [events: ValueOf<S>[], options?: AppendOptions] | [key: K, events: ValueOf<ChildOf<S, K>>[], options?: AppendOptions]
): Promise<AppendResult> {
const hasKey = !Array.isArray(args[0])
const events = args[hasKey ? 1 : 0]
const options = (args[hasKey ? 2 : 1] ?? {}) as AppendOptions
if (!Array.isArray(events)) return Promise.reject(new TypeError('append() expects an array of events'))
return this.withStream(hasKey, args[0], (stream) => this.appendTo(stream, events, options.expectedVersion ?? 'any'))
}
/** `read(options?)` this stream; `read(key, options?)` a child stream. */
async *read<K extends KeyOf<S>>(...args: [options?: ReadOptions] | [key: K, options?: ReadOptions]): AsyncGenerator<StoredEvent<ValueAt<S, K>>> {
const hasKey = typeof args[0] !== 'object' && args.length > 0
const stream = this.streamPath(hasKey, args[0])
yield* this.readFrom(stream, ((hasKey ? args[1] : args[0]) ?? {}) as ReadOptions) as AsyncIterable<StoredEvent<ValueAt<S, K>>>
}
/** Current version of this stream (or a child's); 0 when empty. */
version(key?: KeyOf<S>): Promise<number> {
return this.withStream(arguments.length > 0, key, (stream) => this.versionOf(stream))
}
/** Keys of child streams. */
keys(options: StreamKeysOptions = {}): AsyncIterable<KeyString<S>> {
return this.streamsOf(this.path, options) as AsyncIterable<KeyString<S>>
}
private withStream<T>(hasKey: boolean, key: unknown, fn: (stream: readonly string[]) => Promise<T>): Promise<T> {
try {
return fn(this.streamPath(hasKey, key))
} catch (error) {
return Promise.reject(error)
}
}
private streamPath(hasKey: boolean, key: unknown): readonly string[] {
if (hasKey) return [...this.path, Keys.segment(key)]
if (this.path.length === 0) throw new TypeError('The root node is not a stream')
return this.path
}
}
src/event-storage/event-storage-kv.ts
+151
View File
@@ -0,0 +1,151 @@
import type { BaseKVStorage } from '../kv-storage/base-kv-storage.js'
import { ConcurrencyError } from '../utils/errors.js'
import { Keys } from '../utils/keys.js'
import type { AnySpec, Spec } from '../utils/schema.js'
import { BaseEventStorage, type AppendResult, type ReadOptions, type StoredEvent, type StreamKeysOptions } from './base-event-storage.js'
/** Stored value of one commit key. */
interface Commit {
firstVersion: number
at: number
events: unknown[]
}
interface Head {
commit: number
version: number
}
export interface EventStorageKVOptions {
/** Retries after losing a commit race when `expectedVersion` is `'any'`. Default 5. */
maxRetries?: number
/** Max streams whose head is cached. Default 10 000. */
cacheSize?: number
now?: () => number
}
/**
* Event storage on any KV storage. Each append is one commit key, densely numbered:
* `<stream>/~e/<commit:12>` → `{ firstVersion, at, events }`. Commits are claimed with
* `writeIfAbsent`, so a writer with a stale view always collides with the existing key.
*/
export class EventStorageKV<S extends Spec = AnySpec> extends BaseEventStorage<S> {
private readonly heads = new Map<string, Head>()
private readonly maxRetries: number
private readonly cacheSize: number
private readonly now: () => number
constructor(
private readonly kv: BaseKVStorage,
options: EventStorageKVOptions = {},
) {
super()
this.maxRetries = options.maxRetries ?? 5
this.cacheSize = options.cacheSize ?? 10_000
this.now = options.now ?? Date.now
}
async appendTo(stream: readonly string[], events: unknown[], expectedVersion: number | 'any'): Promise<AppendResult> {
const base = this.commitPrefix(stream)
for (let attempt = 0; ; attempt++) {
let head = this.heads.get(base) ?? (await this.loadHead(base))
// The cache may be stale; only trust a mismatch after re-reading.
if (expectedVersion !== 'any' && head.version !== expectedVersion) head = await this.loadHead(base)
if (expectedVersion !== 'any' && head.version !== expectedVersion) throw new ConcurrencyError(base, expectedVersion, head.version)
if (events.length === 0) return { version: head.version }
const commit: Commit = { firstVersion: head.version + 1, at: this.now(), events }
const [written] = await this.kv.writeIfAbsent([[base + EventStorageKV.pad(head.commit + 1), commit]])
if (written) {
const next = { commit: head.commit + 1, version: head.version + events.length }
this.cacheHead(base, next)
return { version: next.version }
}
this.heads.delete(base)
if (expectedVersion !== 'any' || attempt >= this.maxRetries) {
throw new ConcurrencyError(base, expectedVersion, (await this.loadHead(base)).version)
}
}
}
async *readFrom(stream: readonly string[], options: ReadOptions): AsyncGenerator<StoredEvent<unknown>> {
const from = Math.max(1, options.from ?? 1)
const to = options.to ?? Infinity
let remaining = options.limit ?? Infinity
if (to < from || remaining <= 0) return
const base = this.commitPrefix(stream)
const start = from === 1 ? 1 : await this.findCommit(base, from)
if (start === 0) return
for await (const [, value] of this.kv.scan({ gte: base + EventStorageKV.pad(start), lt: this.commitEnd(base), values: true })) {
const commit = value as Commit
for (let i = 0; i < commit.events.length; i++) {
const version = commit.firstVersion + i
if (version < from) continue
if (version > to || remaining-- <= 0) return
yield { version, at: commit.at, data: commit.events[i] }
}
}
}
async versionOf(stream: readonly string[]): Promise<number> {
return (await this.loadHead(this.commitPrefix(stream))).version
}
async *streamsOf(parent: readonly string[], options: StreamKeysOptions): AsyncGenerator<string> {
yield* parent.reduce<BaseKVStorage>((node, segment) => node.child(segment), this.kv).keys(options)
}
/** Last commit of a stream: one reverse scan with limit 1. Refreshes the cache. */
private async loadHead(base: string): Promise<Head> {
for await (const [key, value] of this.kv.scan({ gte: base, lt: this.commitEnd(base), reverse: true, limit: 1, values: true })) {
const commit = value as Commit
const head = { commit: Number(key.slice(base.length)), version: commit.firstVersion + commit.events.length - 1 }
this.cacheHead(base, head)
return head
}
return { commit: 0, version: 0 }
}
/**
* Commit containing `version`, or 0 if the stream is shorter. Commit `n` starts at a version
* >= n, so it's at most `version`: try that first (exact when commits hold one event each),
* then binary-search the commits below it.
*/
private async findCommit(base: string, version: number): Promise<number> {
const firstVersion = async (commit: number) => ((await this.kv.readMany([base + EventStorageKV.pad(commit)]))[0] as Commit | undefined)?.firstVersion
const guess = await firstVersion(version)
if (guess !== undefined && guess <= version) return version
let hi = version - 1
if (guess === undefined) {
const head = await this.loadHead(base)
if (head.version < version) return 0
hi = head.commit
}
let lo = 1
while (lo < hi) {
const mid = (lo + hi + 1) >>> 1
if ((await firstVersion(mid))! <= version) lo = mid
else hi = mid - 1
}
return lo
}
private cacheHead(base: string, head: Head): void {
this.heads.delete(base)
this.heads.set(base, head)
if (this.heads.size > this.cacheSize) this.heads.delete(this.heads.keys().next().value!)
}
private commitPrefix(stream: readonly string[]): string {
const prefix = this.kv.prefix + Keys.join(stream)
return `${prefix}/~e/`
}
private commitEnd(base: string): string {
return base.slice(0, -1) + Keys.afterSeparator
}
private static pad(commit: number): string {
return String(commit).padStart(12, '0')
}
}
src/events/index.ts
-2
View File
@@ -1,2 +0,0 @@
export * from './types.js'
export * from './kv.js'
src/events/kv.ts
-289
View File
@@ -1,289 +0,0 @@
import { LRU } from '../core/lru.js'
import { KeyedMutex } from '../core/mutex.js'
import { escapeSegment, RESERVED, succ, toSegment } from '../core/path.js'
import { ConcurrencyError } from '../errors.js'
import type { KVDriver } from '../kv/driver.js'
import type { KVNode } from '../kv/node.js'
import type { AnySpec, ChildOf, KeyOf, KeyString, Spec, ValueOf } from '../types/schema.js'
import type { AppendOptions, AppendResult, EventNode, ReadOptions, StoredEvent, StreamKeysOptions } from './types.js'
export interface KVEventStorageOptions {
/** Max streams whose head is cached. Default 10 000. */
headCacheSize?: number
/** Retries after a commit collision when `expectedVersion` is `'any'`. Default 5. */
maxRetries?: number
/** Clock for commit timestamps. Default `Date.now`. */
now?: () => number
}
/** Stored value of one commit key. */
export interface CommitRecord<E = unknown> {
firstVersion: number
at: number
events: E[]
}
interface Head {
commit: number
version: number
}
interface Stream {
/** Stream's own key (escaped); identifies it in the cache and in errors. */
key: string
/** `<key>/~e/` — commit keys are `base + pad12(seq)`. */
base: string
/** Exclusive upper bound of commit keys. */
end: string
}
/** @internal */
interface EventContext {
readonly driver: KVDriver
readonly sep: string
readonly heads: LRU<string, Head>
readonly mutex: KeyedMutex
readonly maxRetries: number
readonly now: () => number
}
const SEQ_WIDTH = 12
const SEARCH_FANOUT = 8
const EMPTY_HEAD: Head = { commit: 0, version: 0 }
const pad = (seq: number) => String(seq).padStart(SEQ_WIDTH, '0')
/**
* Event node over a `KVNode`. Each append is one commit key, densely numbered:
* `<stream>/~e/<commitSeq:12>` → `{ firstVersion, at, events }`. Commits are claimed with
* `setIfAbsent`, so any stale writer collides with the existing key. Without `setIfAbsent`,
* appends are serialised by an in-process per-stream mutex (single-writer only).
*/
export class KVEventNode<S extends Spec = AnySpec> implements EventNode<S> {
readonly #ctx: EventContext
readonly #kv: KVNode<AnySpec>
/** @internal Use `KVEventStorage` and `child()`. */
constructor(ctx: EventContext, kv: KVNode<AnySpec>) {
this.#ctx = ctx
this.#kv = kv
}
get path(): readonly string[] {
return this.#kv.path
}
/** The underlying KV node. */
get kv(): KVNode<AnySpec> {
return this.#kv
}
child<K extends KeyOf<S>>(key: K): KVEventNode<ChildOf<S, K>> {
return new KVEventNode(this.#ctx, this.#kv.child(key as string).as<AnySpec>())
}
as<S2 extends Spec>(): KVEventNode<S2> {
return this as unknown as KVEventNode<S2>
}
append(events: readonly ValueOf<S>[], opts?: AppendOptions): Promise<AppendResult>
append<K extends KeyOf<S>>(key: K, events: readonly ValueOf<ChildOf<S, K>>[], opts?: AppendOptions): Promise<AppendResult>
async append(...args: unknown[]): Promise<AppendResult> {
const self = Array.isArray(args[0])
const stream = self ? this.#self() : this.#child(args[0])
const events = args[self ? 0 : 1]
const opts = (args[self ? 1 : 2] ?? {}) as AppendOptions
if (!Array.isArray(events)) throw new TypeError('append() expects an array of events')
const expected = opts.expectedVersion ?? 'any'
if (expected !== 'any' && !(Number.isInteger(expected) && expected >= 0)) throw new TypeError(`Invalid expectedVersion ${String(expected)}`)
const hint = expected !== 'any' && Number.isInteger(opts.expectedCommit) ? { commit: opts.expectedCommit!, version: expected } : undefined
return this.#append(stream, events, expected, hint)
}
read(opts?: ReadOptions): AsyncIterable<StoredEvent<ValueOf<S>>>
read<K extends KeyOf<S>>(key: K, opts?: ReadOptions): AsyncIterable<StoredEvent<ValueOf<ChildOf<S, K>>>>
async *read(...args: unknown[]): AsyncGenerator<StoredEvent<any>> {
const self = args.length === 0 || (typeof args[0] === 'object' && args[0] !== null)
const stream = self ? this.#self() : this.#child(args[0])
yield* this.#read(stream, ((self ? args[0] : args[1]) ?? {}) as ReadOptions)
}
version(): Promise<number>
version<K extends KeyOf<S>>(key: K): Promise<number>
async version(...args: unknown[]): Promise<number> {
const stream = args.length === 0 ? this.#self() : this.#child(args[0])
return (await this.#loadHead(stream)).version
}
keys(opts: StreamKeysOptions = {}): AsyncIterable<KeyString<S>> {
return this.#kv.keys(opts) as AsyncIterable<KeyString<S>>
}
// ---------------------------------------------------------------- internals
#self(): Stream {
if (this.#kv.isRoot) throw new TypeError('The root node is not a stream')
return this.#streamAt(this.#kv.selfKey)
}
#child(key: unknown): Stream {
return this.#streamAt(this.#kv.prefix + escapeSegment(toSegment(key), this.#ctx.sep))
}
#streamAt(key: string): Stream {
const sep = this.#ctx.sep
const head = key + sep + RESERVED.events
return { key, base: head + sep, end: head + succ(sep) }
}
async #loadHead(s: Stream): Promise<Head> {
for await (const [key, value] of this.#ctx.driver.scan({ gte: s.base, lt: s.end, reverse: true, limit: 1, values: true })) {
const c = value as CommitRecord
const head = { commit: Number(key.slice(s.base.length)), version: c.firstVersion + c.events.length - 1 }
this.#ctx.heads.set(s.key, head)
return head
}
return EMPTY_HEAD
}
async #append(s: Stream, events: readonly unknown[], expected: number | 'any', hint?: Head): Promise<AppendResult> {
const { driver, heads, maxRetries } = this.#ctx
if (events.length === 0) {
const head = await this.#loadHead(s)
if (expected !== 'any' && head.version !== expected) throw new ConcurrencyError(s.key, expected, head.version)
return { version: head.version, commit: head.commit }
}
if (!driver.setIfAbsent) return this.#ctx.mutex.run(s.key, () => this.#appendSingleWriter(s, events, expected))
for (let attempt = 0; ; attempt++) {
let head = heads.get(s.key) ?? (attempt === 0 ? hint : undefined)
if (!head) head = await this.#loadHead(s)
else if (expected !== 'any' && head.version !== expected) head = await this.#loadHead(s) // cache may be stale
if (expected !== 'any' && head.version !== expected) throw new ConcurrencyError(s.key, expected, head.version)
const seq = head.commit + 1
const record: CommitRecord = { firstVersion: head.version + 1, at: this.#ctx.now(), events: [...events] }
const [written] = await driver.setIfAbsent([[s.base + pad(seq), record]])
if (written) {
const next = { commit: seq, version: head.version + events.length }
heads.set(s.key, next)
return next
}
// Lost the race for this commit number.
heads.delete(s.key)
if (expected !== 'any' || attempt >= maxRetries) {
throw new ConcurrencyError(s.key, expected, (await this.#loadHead(s)).version)
}
}
}
async #appendSingleWriter(s: Stream, events: readonly unknown[], expected: number | 'any'): Promise<AppendResult> {
const head = this.#ctx.heads.get(s.key) ?? (await this.#loadHead(s))
if (expected !== 'any' && head.version !== expected) throw new ConcurrencyError(s.key, expected, head.version)
const seq = head.commit + 1
const record: CommitRecord = { firstVersion: head.version + 1, at: this.#ctx.now(), events: [...events] }
await this.#ctx.driver.setMany([[s.base + pad(seq), record]])
const next = { commit: seq, version: head.version + events.length }
this.#ctx.heads.set(s.key, next)
return next
}
/**
* Sequence number of the commit containing `version`, or 0 if the stream hasn't reached it.
* Commit `n` has `firstVersion >= n`, so the answer is at most `version`: probe that first
* (one round trip when commits hold single events), then k-ary search with `getMany` probes.
*/
async #locate(s: Stream, version: number): Promise<number> {
const { driver } = this.#ctx
const [guess] = (await driver.getMany([s.base + pad(version)])) as (CommitRecord | undefined)[]
if (guess && guess.firstVersion <= version) return version
let hi: number
if (guess) hi = version - 1
else {
const head = await this.#loadHead(s)
if (head.version < version) return 0
hi = head.commit
}
let lo = 1 // commit 1 starts at version 1
while (lo < hi) {
const probes: number[] = []
for (let i = 1; i <= SEARCH_FANOUT; i++) {
const p = lo + Math.ceil(((hi - lo) * i) / SEARCH_FANOUT)
if (p !== probes.at(-1)) probes.push(p)
}
const found = (await driver.getMany(probes.map((p) => s.base + pad(p)))) as (CommitRecord | undefined)[]
for (let i = 0; i < probes.length; i++) {
const c = found[i]
if (c && c.firstVersion <= version) lo = probes[i]!
else {
hi = probes[i]! - 1
break
}
}
}
return lo
}
async *#read(s: Stream, opts: ReadOptions): AsyncGenerator<StoredEvent<unknown>> {
const from = Math.max(1, opts.from ?? 1)
const to = opts.to ?? Infinity
let remaining = opts.limit ?? Infinity
if (to < from || remaining <= 0) return
const { driver } = this.#ctx
if (!opts.reverse) {
let gte = s.base
if (from > 1) {
const seq = await this.#locate(s, from)
if (seq === 0) return
gte = s.base + pad(seq)
}
for await (const [key, value] of driver.scan({ gte, lt: s.end, values: true })) {
const c = value as CommitRecord
const commit = Number(key.slice(s.base.length))
for (let i = 0; i < c.events.length; i++) {
const version = c.firstVersion + i
if (version < from) continue
if (version > to) return
yield { version, commit, at: c.at, data: c.events[i] }
if (--remaining <= 0) return
}
}
return
}
let lt = s.end
if (to !== Infinity) {
const seq = await this.#locate(s, to)
if (seq !== 0) lt = s.base + pad(seq + 1)
}
for await (const [key, value] of driver.scan({ gte: s.base, lt, reverse: true, values: true })) {
const c = value as CommitRecord
const commit = Number(key.slice(s.base.length))
for (let i = c.events.length - 1; i >= 0; i--) {
const version = c.firstVersion + i
if (version > to) continue
if (version < from) return
yield { version, commit, at: c.at, data: c.events[i] }
if (--remaining <= 0) return
}
}
}
}
/** Event storage over a `KVStorage` (or any `KVNode` as root). */
export class KVEventStorage<S extends Spec = AnySpec> extends KVEventNode<S> {
constructor(kv: KVNode<AnySpec>, opts: KVEventStorageOptions = {}) {
super(
{
driver: kv.driver,
sep: kv.separator,
heads: new LRU(opts.headCacheSize ?? 10_000),
mutex: new KeyedMutex(),
maxRetries: opts.maxRetries ?? 5,
now: opts.now ?? Date.now,
},
kv,
)
}
}
src/events/types.ts
-69
View File
@@ -1,69 +0,0 @@
import type { AnySpec, ChildOf, KeyOf, KeyString, Spec, ValueOf } from '../types/schema.js'
export interface AppendOptions {
/** Version the stream must be at (0 = empty). Default `'any'`. */
expectedVersion?: number | 'any'
/**
* Commit sequence of the head at `expectedVersion`, if the caller knows it (e.g. from a
* projection). Lets a cold append skip the head lookup. Must come from the stream itself: a
* stale hint only causes a collision, but an invented one could leave a gap. Ignored unless
* `expectedVersion` is a number; backends may ignore it.
*/
expectedCommit?: number
}
export interface AppendResult {
/** Stream version after the append (the last event's version). */
version: number
/** Commit sequence number of this append (the current head commit when no events were given). */
commit: number
}
export interface ReadOptions {
/** First version to read (inclusive). Default 1. */
from?: number
/** Last version to read (inclusive). Default: the head. */
to?: number
reverse?: boolean
limit?: number
}
export interface StoredEvent<E> {
version: number
commit: number
/** Commit timestamp (ms since epoch). */
at: number
data: E
}
export interface StreamKeysOptions {
limit?: number
after?: string | number
reverse?: boolean
}
/**
* One node of an event store: a stream (its own events) and a container of child streams.
* Same navigation and self/child arity rules as `KVNode`.
*/
export interface EventNode<S extends Spec = AnySpec> {
readonly path: readonly string[]
child<K extends KeyOf<S>>(key: K): EventNode<ChildOf<S, K>>
as<S2 extends Spec>(): EventNode<S2>
append(events: readonly ValueOf<S>[], opts?: AppendOptions): Promise<AppendResult>
append<K extends KeyOf<S>>(key: K, events: readonly ValueOf<ChildOf<S, K>>[], opts?: AppendOptions): Promise<AppendResult>
read(opts?: ReadOptions): AsyncIterable<StoredEvent<ValueOf<S>>>
read<K extends KeyOf<S>>(key: K, opts?: ReadOptions): AsyncIterable<StoredEvent<ValueOf<ChildOf<S, K>>>>
/** Current version (0 = empty). Always reads storage. */
version(): Promise<number>
version<K extends KeyOf<S>>(key: K): Promise<number>
/** Child stream keys. */
keys(opts?: StreamKeysOptions): AsyncIterable<KeyString<S>>
}
/** The root of an event store. Native backends (EventStoreDB, Kafka, …) implement this too. */
export type EventStorage<S extends Spec = AnySpec> = EventNode<S>
src/http/client.ts
-115
View File
@@ -1,115 +0,0 @@
import { BaseKVDriver, JsonCodec, type BatchOp, type Codec, type ScanOptions } from '../kv/driver.js'
import type { HttpCapabilities } from './handler.js'
export interface HttpKVDriverOptions {
/** Base URL of a `createKVHttpHandler` endpoint, e.g. `https://host/kv`. */
url: string
/** Optional capabilities the server's driver supports. Default: all. `connect()` discovers them. */
capabilities?: Partial<HttpCapabilities>
codec?: Codec
headers?: Record<string, string>
fetch?: typeof fetch
}
/**
* Driver client for `createKVHttpHandler`. Optional methods exist only when the server
* supports them, so stores' feature detection works across HTTP.
*/
export class HttpKVDriver extends BaseKVDriver {
readonly #url: string
readonly #codec: Codec
readonly #headers: Record<string, string>
readonly #fetch: typeof fetch
readonly #nativeDeleteRange: boolean
setIfAbsent?: (entries: [string, unknown][]) => Promise<boolean[]>
batch?: (ops: BatchOp[]) => Promise<void>
/** Creates a client after asking the server which optional capabilities it has. */
static async connect(opts: Omit<HttpKVDriverOptions, 'capabilities'>): Promise<HttpKVDriver> {
const probe = new HttpKVDriver({ ...opts, capabilities: {} })
const capabilities = (await probe.#request('GET', '/capabilities')) as HttpCapabilities
return new HttpKVDriver({ ...opts, capabilities })
}
constructor(opts: HttpKVDriverOptions) {
super()
this.#url = opts.url.replace(/\/$/, '')
this.#codec = opts.codec ?? JsonCodec
this.#headers = opts.headers ?? {}
this.#fetch = opts.fetch ?? globalThis.fetch.bind(globalThis)
const caps = { setIfAbsent: true, batch: true, deleteRange: true, ...opts.capabilities }
this.#nativeDeleteRange = caps.deleteRange
if (caps.setIfAbsent) this.setIfAbsent = async (entries) => ((await this.#request('POST', '/setIfAbsent', { entries })) as { written: boolean[] }).written
if (caps.batch) this.batch = async (ops) => void (await this.#request('POST', '/batch', { ops }))
}
async getMany(keys: string[]): Promise<unknown[]> {
if (keys.length === 0) return []
const { values, missing } = (await this.#request('POST', '/getMany', { keys })) as { values: unknown[]; missing: number[] }
for (const i of missing) values[i] = undefined
return values
}
async setMany(entries: [string, unknown][]): Promise<void> {
if (entries.length) await this.#request('POST', '/setMany', { entries })
}
async deleteMany(keys: string[]): Promise<void> {
if (keys.length) await this.#request('POST', '/deleteMany', { keys })
}
override async deleteRange(gte: string, lt: string): Promise<void> {
if (!this.#nativeDeleteRange) return super.deleteRange(gte, lt)
await this.#request('POST', '/deleteRange', { gte, lt })
}
async *scan(opts: ScanOptions): AsyncGenerator<[string, unknown]> {
if (opts.limit !== undefined && opts.limit <= 0) return
const q = new URLSearchParams()
if (opts.gte !== undefined) q.set('gte', opts.gte)
if (opts.lt !== undefined) q.set('lt', opts.lt)
if (opts.limit !== undefined && opts.limit !== Infinity) q.set('limit', String(opts.limit))
if (opts.reverse) q.set('reverse', 'true')
if (opts.values) q.set('values', 'true')
const res = await this.#fetch(`${this.#url}/scan?${q}`, { headers: this.#headers })
if (!res.ok || !res.body) throw await httpError(res)
const reader = res.body.getReader()
const decoder = new TextDecoder()
let buf = ''
try {
for (;;) {
const { done, value } = await reader.read()
if (done) break
buf += decoder.decode(value, { stream: true })
let nl: number
while ((nl = buf.indexOf('\n')) !== -1) {
const line = buf.slice(0, nl)
buf = buf.slice(nl + 1)
const [key, v] = this.#codec.decode(line) as [string, unknown?]
yield [key, v]
}
}
} finally {
await reader.cancel().catch(() => {})
}
}
async #request(method: 'GET' | 'POST', path: string, body?: unknown): Promise<unknown> {
const res = await this.#fetch(this.#url + path, {
method,
headers: { ...this.#headers, ...(body === undefined ? {} : { 'content-type': 'application/json' }) },
body: body === undefined ? undefined : this.#codec.encode(body),
})
if (!res.ok) throw await httpError(res)
return res.status === 204 ? undefined : this.#codec.decode(await res.text())
}
}
async function httpError(res: Response): Promise<Error> {
let detail = ''
try {
detail = await res.text()
} catch {}
return new Error(`HTTP ${res.status}: ${detail}`)
}
src/http/handler.ts
-115
View File
@@ -1,115 +0,0 @@
import { JsonCodec, type BatchOp, type Codec, type KVDriver, type ScanOptions } from '../kv/driver.js'
export interface KVHttpHandlerOptions {
/** Serialises request/response bodies and NDJSON scan lines. Must not emit newlines. Default JSON. */
codec?: Codec
/** Path prefix to strip, e.g. `/kv`. Default `''`. */
basePath?: string
}
export interface HttpCapabilities {
setIfAbsent: boolean
batch: boolean
deleteRange: boolean
}
/**
* Framework-agnostic HTTP front for a driver (`Request → Response`).
*
* - `POST /getMany {keys}` → `{values, missing}` (`missing` = indices of absent keys)
* - `POST /setMany {entries}`, `POST /deleteMany {keys}`, `POST /batch {ops}` → 204
* - `POST /setIfAbsent {entries}` → `{written}`
* - `POST /deleteRange {gte, lt}` → 204
* - `GET /scan?gte&lt&limit&reverse&values` → NDJSON, one `[key]` or `[key, value]` per line
* - `GET /capabilities` → `{setIfAbsent, batch, deleteRange}`
*/
export function createKVHttpHandler(driver: KVDriver, opts: KVHttpHandlerOptions = {}): (req: Request) => Promise<Response> {
const codec = opts.codec ?? JsonCodec
const base = opts.basePath ?? ''
const reply = (body: unknown, status = 200) => new Response(codec.encode(body), { status, headers: { 'content-type': 'application/json' } })
const noContent = () => new Response(null, { status: 204 })
const fail = (status: number, message: string) => reply({ error: message }, status)
return async (req) => {
const url = new URL(req.url)
if (!url.pathname.startsWith(base)) return fail(404, 'Not found')
const route = `${req.method} ${url.pathname.slice(base.length)}`
try {
const body = async <T>() => codec.decode(await req.text()) as T
switch (route) {
case 'POST /getMany': {
const { keys } = await body<{ keys: string[] }>()
const found = await driver.getMany(keys)
const missing: number[] = []
const values = found.map((v, i) => (v === undefined ? (missing.push(i), null) : v))
return reply({ values, missing })
}
case 'POST /setMany':
await driver.setMany((await body<{ entries: [string, unknown][] }>()).entries)
return noContent()
case 'POST /deleteMany':
await driver.deleteMany((await body<{ keys: string[] }>()).keys)
return noContent()
case 'POST /setIfAbsent':
if (!driver.setIfAbsent) return fail(501, 'setIfAbsent not supported')
return reply({ written: await driver.setIfAbsent((await body<{ entries: [string, unknown][] }>()).entries) })
case 'POST /batch':
if (!driver.batch) return fail(501, 'batch not supported')
await driver.batch((await body<{ ops: BatchOp[] }>()).ops)
return noContent()
case 'POST /deleteRange': {
if (!driver.deleteRange) return fail(501, 'deleteRange not supported')
const { gte, lt } = await body<{ gte: string; lt: string }>()
await driver.deleteRange(gte, lt)
return noContent()
}
case 'GET /capabilities':
return reply({ setIfAbsent: !!driver.setIfAbsent, batch: !!driver.batch, deleteRange: !!driver.deleteRange } satisfies HttpCapabilities)
case 'GET /scan':
return scanResponse(driver, parseScan(url.searchParams), codec)
default:
return fail(404, `No route ${route}`)
}
} catch (err) {
return fail(500, err instanceof Error ? err.message : String(err))
}
}
}
function parseScan(q: URLSearchParams): ScanOptions {
const opts: ScanOptions = {}
if (q.has('gte')) opts.gte = q.get('gte')!
if (q.has('lt')) opts.lt = q.get('lt')!
if (q.has('limit')) opts.limit = Number(q.get('limit'))
if (q.get('reverse') === 'true') opts.reverse = true
if (q.get('values') === 'true') opts.values = true
return opts
}
function scanResponse(driver: KVDriver, opts: ScanOptions, codec: Codec): Response {
const it = driver.scan(opts)[Symbol.asyncIterator]()
const encoder = new TextEncoder()
// Chunks grow so short (skip-scan) reads don't make the server run far ahead.
let chunk = 16
const stream = new ReadableStream<Uint8Array>({
async pull(controller) {
let buf = ''
for (let i = 0; i < chunk; i++) {
const r = await it.next()
if (r.done) {
if (buf) controller.enqueue(encoder.encode(buf))
controller.close()
return
}
const [key, value] = r.value
buf += codec.encode(opts.values ? [key, value] : [key]) + '\n'
}
chunk = Math.min(chunk * 2, 256)
controller.enqueue(encoder.encode(buf))
},
async cancel() {
await it.return?.()
},
})
return new Response(stream, { headers: { 'content-type': 'application/x-ndjson' } })
}
src/http/index.ts
-2
View File
@@ -1,2 +0,0 @@
export * from './handler.js'
export * from './client.js'
src/index.ts
+18 -8
View File
@@ -1,8 +1,18 @@
export * from './errors.js'
export * from './core/path.js'
export * from './core/encoding.js'
export * from './kv/index.js'
export * from './types/schema.js'
export * from './events/index.js'
export * from './documents/index.js'
export * from './http/index.js'
export * from './kv-storage/base-kv-storage.js'
export * from './kv-storage/kv-storage-in-memory.js'
export * from './kv-storage/kv-storage-http.js'
export * from './kv-storage/kv-storage-http-server.js'
export * from './event-storage/base-event-storage.js'
export * from './event-storage/event-storage-kv.js'
export * from './document-storage/base-document-storage.js'
export * from './document-storage/document-storage-in-memory.js'
export * from './document-storage/document-storage-kv.js'
export * from './document-storage/event-sourced-document-storage.js'
export * from './utils/document-path.js'
export * from './utils/document-query.js'
export * from './utils/document-update.js'
export * from './utils/errors.js'
export * from './utils/json.js'
export * from './utils/keys.js'
export * from './utils/schema.js'
export * from './utils/ulid.js'
src/kv-storage/base-kv-storage.ts
+184
View File
@@ -0,0 +1,184 @@
import { Keys } from '../utils/keys.js'
import type { AnySpec, ChildOf, EntryOf, KeyOf, KeyString, Spec, StringEntryOf, ValueAt, ValueOf } from '../utils/schema.js'
export interface ScanOptions {
/** Inclusive lower bound. */
gte?: string
/** Exclusive upper bound. */
lt?: string
limit?: number
reverse?: boolean
/** Include values; otherwise they are `undefined`. */
values?: boolean
}
export interface KeysOptions {
limit?: number
/** Start after this child key (exclusive), in the direction of travel. */
after?: string | number
reverse?: boolean
}
/**
* Ordered key-value storage. Implementations provide five raw primitives over absolute keys;
* everything else (navigation, self/child reads and writes, child listing) is defined here once.
*
* Every instance is a node: `child(key)` returns a view of the same storage rooted at that key.
* Methods taking an optional `key` act on that child, or on this node's own value when omitted.
* A key that is passed but `undefined`/`null` throws rather than silently meaning "self".
*/
export abstract class BaseKVStorage<S extends Spec = AnySpec> {
/** Unescaped path of this node (`[]` for the root). */
path: readonly string[] = []
/** This node's own key (`''` for the root). */
key = ''
/** Prefix of every descendant key (`''` for the root). */
prefix = ''
/** Values for absolute keys, in order; `undefined` for missing. */
abstract readMany(keys: string[]): Promise<unknown[]>
abstract writeMany(entries: [key: string, value: unknown][]): Promise<void>
abstract removeMany(keys: string[]): Promise<void>
/** Writes each entry only if its key is absent; `true` = written. Must be atomic per key. */
abstract writeIfAbsent(entries: [key: string, value: unknown][]): Promise<boolean[]>
/** Entries in key order (descending when `reverse`). */
abstract scan(options?: ScanOptions): AsyncIterable<[key: string, value: unknown]>
// ------------------------------------------------------------------ navigation
/** A view of this storage rooted at `key`. Synchronous; no I/O. */
child<K extends KeyOf<S>>(key: K): BaseKVStorage<ChildOf<S, K>> {
const node = Object.create(this) as BaseKVStorage<ChildOf<S, K>>
node.path = [...this.path, Keys.segment(key)]
node.key = this.prefix + Keys.escape(Keys.segment(key))
node.prefix = node.key + Keys.separator
return node
}
/** Re-types this node. No runtime effect. */
as<S2 extends Spec>(): BaseKVStorage<S2> {
return this as unknown as BaseKVStorage<S2>
}
// ------------------------------------------------------------------ reads and writes
async get<K extends KeyOf<S> | undefined = undefined>(key?: K): Promise<ValueAt<S, K> | undefined> {
const [value] = await this.readMany([this.keyOf(arguments.length > 0, key)])
return value as ValueAt<S, K> | undefined
}
async getMany<K extends KeyOf<S>>(keys: readonly K[]): Promise<(ValueOf<ChildOf<S, K>> | undefined)[]> {
return (await this.readMany(keys.map((k) => this.keyOf(true, k)))) as (ValueOf<ChildOf<S, K>> | undefined)[]
}
/** `set(value)` writes this node's own value; `set(key, value)` a child's. */
async set<K extends KeyOf<S>>(...args: [value: ValueOf<S>] | [key: K, value: ValueOf<ChildOf<S, K>>]): Promise<void> {
const value = args[args.length - 1]
if (value === undefined) throw new TypeError('Cannot set undefined; use delete()')
await this.writeMany([[this.keyOf(args.length > 1, args[0]), value]])
}
async setMany(entries: readonly EntryOf<S>[]): Promise<void> {
if (entries.some(([, v]) => v === undefined)) throw new TypeError('Cannot set undefined; use delete()')
await this.writeMany(entries.map(([k, v]) => [this.keyOf(true, k), v]))
}
/** Deletes this node's own value (or a child's), not descendants. */
async delete(key?: KeyOf<S>): Promise<void> {
await this.removeMany([this.keyOf(arguments.length > 0, key)])
}
async deleteMany(keys: readonly KeyOf<S>[]): Promise<void> {
await this.removeMany(keys.map((k) => this.keyOf(true, k)))
}
/** Deletes this node's own value (or a child's) and all descendants. */
async deleteTree(key?: KeyOf<S>): Promise<void> {
const own = this.keyOf(arguments.length > 0, key, true)
if (own !== '') await this.removeMany([own])
const prefix = own === '' ? '' : own + Keys.separator
for (;;) {
const batch: string[] = []
for await (const [k] of this.scan({ gte: prefix, lt: own === '' ? undefined : own + Keys.afterSeparator, limit: 1000 })) batch.push(k)
if (batch.length === 0) return
await this.removeMany(batch)
}
}
// ------------------------------------------------------------------ listing
/**
* Direct child keys in key order, including children that only have descendants.
* Skip-scan: one seek per child that has descendants, not one read per descendant.
*/
async *keys(options: KeysOptions = {}): AsyncGenerator<KeyString<S>> {
for await (const [key] of this.children(options, false)) yield key as KeyString<S>
}
/** Direct children that have their own value, as `[key, value]`. */
async *entries(options: KeysOptions = {}): AsyncGenerator<StringEntryOf<S>> {
let remaining = options.limit ?? Infinity
for await (const [key, value, hasValue] of this.children({ ...options, limit: undefined }, true)) {
if (remaining <= 0) return
if (!hasValue) continue
remaining--
yield [key, value] as StringEntryOf<S>
}
}
private async *children(options: KeysOptions, values: boolean): AsyncGenerator<[key: string, value: unknown, hasValue: boolean]> {
const { reverse = false } = options
let remaining = options.limit ?? Infinity
let gte = this.prefix
let lt = this.key === '' ? undefined : this.key + Keys.afterSeparator
if (options.after !== undefined) {
const after = this.prefix + Keys.escape(Keys.segment(options.after))
if (reverse) lt = after
else gte = after + Keys.afterSeparator
}
let last: string | undefined
let restart = true
let iterator: AsyncIterator<[string, unknown]> | undefined
try {
while (remaining > 0) {
if (restart) {
await iterator?.return?.()
iterator = this.scan({ gte, lt, reverse, values })[Symbol.asyncIterator]()
restart = false
}
const next = await iterator!.next()
if (next.done) return
const [key, value] = next.value
const end = key.indexOf(Keys.separator, this.prefix.length)
const segment = key.slice(this.prefix.length, end === -1 ? undefined : end)
const isOwn = end === -1
if (segment !== last && !Keys.isReserved(segment)) {
// Forward, a child's own key comes before its descendants; in reverse it comes after
// them and is skipped by the jump below, so read it directly when values are wanted.
let own = isOwn ? value : undefined
if (!isOwn && reverse && values) [own] = await this.readMany([this.prefix + segment])
yield [Keys.unescape(segment), own, isOwn || own !== undefined]
remaining--
}
last = segment
// Jump over the rest of this child's subtree instead of reading it.
if (!isOwn || Keys.isReserved(segment)) {
if (reverse) lt = this.prefix + segment
else gte = this.prefix + segment + Keys.afterSeparator
restart = true
}
}
} finally {
await iterator?.return?.()
}
}
// ------------------------------------------------------------------ internals
/** Absolute key for a child (`hasKey`) or this node; validates keys. */
protected keyOf(hasKey: boolean, key: unknown, allowRoot = false): string {
if (hasKey) return this.prefix + Keys.escape(Keys.segment(key))
if (this.key === '' && !allowRoot) throw new TypeError('The root node has no own value')
return this.key
}
}
src/kv-storage/kv-storage-http-server.ts
+70
View File
@@ -0,0 +1,70 @@
import type { BaseKVStorage, ScanOptions } from './base-kv-storage.js'
/**
* Serves any KV storage over HTTP for `KVStorageHttp`. Framework-agnostic: `handle(Request)`
* returns a `Response`. All routes are `POST /<primitive>` with a JSON body; `scan` streams NDJSON.
* The raw primitives use absolute keys, so this exposes the whole store behind `storage`.
*/
export class KVStorageHttpServer {
constructor(
private readonly storage: BaseKVStorage,
private readonly basePath = '',
) {}
async handle(request: Request): Promise<Response> {
const path = new URL(request.url).pathname
if (request.method !== 'POST' || !path.startsWith(this.basePath)) return KVStorageHttpServer.error(404, 'Not found')
try {
const body = (await request.json()) as { keys: string[]; entries: [string, unknown][] } & ScanOptions
switch (path.slice(this.basePath.length)) {
case '/readMany': {
const values = await this.storage.readMany(body.keys)
const missing = values.flatMap((v, i) => (v === undefined ? [i] : []))
return Response.json({ values: values.map((v) => v ?? null), missing })
}
case '/writeMany':
await this.storage.writeMany(body.entries)
return new Response(null, { status: 204 })
case '/removeMany':
await this.storage.removeMany(body.keys)
return new Response(null, { status: 204 })
case '/writeIfAbsent':
return Response.json(await this.storage.writeIfAbsent(body.entries))
case '/scan':
return this.scan(body)
default:
return KVStorageHttpServer.error(404, 'Not found')
}
} catch (error) {
return KVStorageHttpServer.error(500, error instanceof Error ? error.message : String(error))
}
}
private scan(options: ScanOptions): Response {
const iterator = this.storage.scan(options)[Symbol.asyncIterator]()
const encoder = new TextEncoder()
const stream = new ReadableStream<Uint8Array>({
async pull(controller) {
// Send in small batches so short reads (skip-scan seeks) don't make the server run far ahead.
let chunk = ''
for (let i = 0; i < 32; i++) {
const next = await iterator.next()
if (next.done) {
if (chunk) controller.enqueue(encoder.encode(chunk))
return controller.close()
}
chunk += JSON.stringify(options.values ? next.value : [next.value[0]]) + '\n'
}
controller.enqueue(encoder.encode(chunk))
},
async cancel() {
await iterator.return?.()
},
})
return new Response(stream, { headers: { 'content-type': 'application/x-ndjson' } })
}
private static error(status: number, message: string): Response {
return Response.json({ error: message }, { status })
}
}
src/kv-storage/kv-storage-http.ts
+79
View File
@@ -0,0 +1,79 @@
import type { AnySpec, Spec } from '../utils/schema.js'
import { BaseKVStorage, type ScanOptions } from './base-kv-storage.js'
export interface KVStorageHttpOptions {
/** Base URL of a `KVStorageHttpServer`, e.g. `https://host/kv`. */
url: string
headers?: Record<string, string>
fetch?: typeof fetch
}
/** KV storage client for a `KVStorageHttpServer`. Same API as any other KV storage. */
export class KVStorageHttp<S extends Spec = AnySpec> extends BaseKVStorage<S> {
private readonly url: string
private readonly headers: Record<string, string>
private readonly fetcher: typeof fetch
constructor(options: KVStorageHttpOptions) {
super()
this.url = options.url.replace(/\/$/, '')
this.headers = options.headers ?? {}
this.fetcher = options.fetch ?? globalThis.fetch.bind(globalThis)
}
async readMany(keys: string[]): Promise<unknown[]> {
if (keys.length === 0) return []
const { values, missing } = (await this.post('readMany', { keys })) as { values: unknown[]; missing: number[] }
for (const i of missing) values[i] = undefined
return values
}
async writeMany(entries: [string, unknown][]): Promise<void> {
if (entries.length) await this.post('writeMany', { entries })
}
async removeMany(keys: string[]): Promise<void> {
if (keys.length) await this.post('removeMany', { keys })
}
async writeIfAbsent(entries: [string, unknown][]): Promise<boolean[]> {
return entries.length ? ((await this.post('writeIfAbsent', { entries })) as boolean[]) : []
}
async *scan(options: ScanOptions = {}): AsyncGenerator<[string, unknown]> {
if (options.limit !== undefined && options.limit <= 0) return
const response = await this.request('scan', options)
const reader = response.body!.pipeThrough(new TextDecoderStream()).getReader()
let buffer = ''
try {
for (;;) {
const { done, value } = await reader.read()
if (done) return
buffer += value
const lines = buffer.split('\n')
buffer = lines.pop()!
for (const line of lines) {
const [key, value] = JSON.parse(line) as [string, unknown?]
yield [key, value]
}
}
} finally {
await reader.cancel().catch(() => {})
}
}
private async post(path: string, body: unknown): Promise<unknown> {
const response = await this.request(path, body)
return response.status === 204 ? undefined : response.json()
}
private async request(path: string, body: unknown): Promise<Response> {
const response = await this.fetcher(`${this.url}/${path}`, {
method: 'POST',
headers: { ...this.headers, 'content-type': 'application/json' },
body: JSON.stringify(body),
})
if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`)
return response
}
}
src/kv-storage/kv-storage-in-memory.ts
+34
View File
@@ -0,0 +1,34 @@
import { Json } from '../utils/json.js'
import { SortedMap } from '../utils/sorted-map.js'
import type { AnySpec, Spec } from '../utils/schema.js'
import { BaseKVStorage, type ScanOptions } from './base-kv-storage.js'
/** In-memory KV storage. Values are deep-copied on write and read. */
export class KVStorageInMemory<S extends Spec = AnySpec> extends BaseKVStorage<S> {
private readonly data = new SortedMap<unknown>()
async readMany(keys: string[]): Promise<unknown[]> {
return keys.map((k) => Json.clone(this.data.get(k)))
}
async writeMany(entries: [string, unknown][]): Promise<void> {
for (const [k, v] of entries) this.data.set(k, Json.clone(v))
}
async removeMany(keys: string[]): Promise<void> {
for (const k of keys) this.data.delete(k)
}
async writeIfAbsent(entries: [string, unknown][]): Promise<boolean[]> {
return entries.map(([k, v]) => !this.data.has(k) && (this.data.set(k, Json.clone(v)), true))
}
async *scan(options: ScanOptions = {}): AsyncGenerator<[string, unknown]> {
let remaining = options.limit ?? Infinity
if (remaining <= 0) return
for (const key of this.data.keys(options.gte, options.lt, options.reverse)) {
yield [key, options.values ? Json.clone(this.data.get(key)) : undefined]
if (--remaining <= 0) return
}
}
}
src/kv/children.ts
-236
View File
@@ -1,236 +0,0 @@
import { escapeSegment, succ } from '../core/path.js'
import type { KVDriver } from './driver.js'
/**
* Direct-children skip-scan, shared by every layer.
*
* ## Child order
*
* Children are ordered by **escaped segment followed by the separator** ("subtree order": the
* order in which their descendants appear in key space). For segments without characters that
* sort below the separator this is plain lexicographic order. When one segment extends another
* with such a character (`a` vs `a-1` with `/`), the longer one comes first (`a-1/` < `a/`).
*
* This is the one order that can be produced without extra reads: a child's own key
* (`p/a`) sorts *before* siblings that extend it (`p/a-1`), while its descendants (`p/a/…`) sort
* after them. The forward scan therefore defers yielding a child seen only through its own key
* until the scan passes `p/a/`, using a stack bounded by segment length.
*
* ## Cost
*
* - Forward: one seek per child that has descendants (to jump over its subtree), one at the start,
* and one per reserved (`~…`) region. Siblings without descendants share an iterator.
* `after` adds one `getMany` when the `after` segment contains a below-separator character.
* - Reverse: as forward, plus one `getMany` per newly found child whose segment contains a
* below-separator character (to find shorter siblings that only have an own key), or that was
* found through a descendant when `own` is requested.
*/
export interface ChildHit {
/** Escaped segment. */
esc: string
/** Whether the child has an own value. Only accurate when `own` was requested. */
hasOwn: boolean
/** The own value, when `values` was requested and `hasOwn`. */
value: unknown
}
export interface ChildScanOptions {
/** Unescaped segment to start after (exclusive), in the direction of travel. */
after?: string
reverse?: boolean
limit?: number
/** Fetch own values. Implies `own`. */
values?: boolean
/** Require accurate `hasOwn`. */
own?: boolean
}
const RESERVED_START = '~'
const RESERVED_END = succ('~')
export async function* scanChildren(
driver: KVDriver,
prefix: string,
end: string | undefined,
sep: string,
opts: ChildScanOptions = {},
): AsyncGenerator<ChildHit> {
const limit = opts.limit ?? Infinity
if (limit <= 0) return
if (opts.reverse) yield* reverseScan(driver, prefix, end, sep, opts, limit)
else yield* forwardScan(driver, prefix, end, sep, opts, limit)
}
/** Prefixes `E[0..j)` (j ≥ 1) where `E[j]` sorts below the separator. */
function lowPrefixes(esc: string, sepCode: number): string[] {
let out: string[] | undefined
for (let j = 1; j < esc.length; j++) {
if (esc.charCodeAt(j) < sepCode) (out ??= []).push(esc.slice(0, j))
}
return out ?? NONE
}
const NONE: string[] = []
async function* forwardScan(
driver: KVDriver,
prefix: string,
end: string | undefined,
sep: string,
opts: ChildScanOptions,
limit: number,
): AsyncGenerator<ChildHit> {
const sepCode = sep.charCodeAt(0)
const succSep = succ(sep)
const values = opts.values === true
const pending: ChildHit[] = []
let emitted = 0
let gte = prefix
if (opts.after !== undefined) {
const escA = escapeSegment(opts.after, sep)
gte = prefix + escA + succSep
// Shorter siblings that extend into `after` with a low char come after it, but their own
// keys sort before `gte`; probe for them.
const cands = lowPrefixes(escA, sepCode)
if (cands.length) {
const found = await driver.getMany(cands.map((g) => prefix + g))
for (let i = 0; i < cands.length; i++) {
if (found[i] !== undefined) pending.push({ esc: cands[i]!, hasOwn: true, value: values ? found[i] : undefined })
}
}
}
const prefixLen = prefix.length
let it = driver.scan({ gte, lt: end, values })[Symbol.asyncIterator]()
const reseek = async (from: string) => {
await it.return?.()
it = driver.scan({ gte: from, lt: end, values })[Symbol.asyncIterator]()
}
try {
outer: for (;;) {
const r = await it.next()
if (r.done) {
while (pending.length) {
yield pending.pop()!
if (++emitted >= limit) return
}
return
}
const key = r.value[0]
const cut = key.indexOf(sep, prefixLen)
const esc = cut === -1 ? key.slice(prefixLen) : key.slice(prefixLen, cut)
while (pending.length) {
const top = pending[pending.length - 1]!
if (top.esc === esc) {
// First descendant of a deferred child: emit it and jump over its subtree.
pending.pop()
yield top
if (++emitted >= limit) return
await reseek(prefix + esc + succSep)
continue outer
}
if (top.esc + sep < esc + sep) {
pending.pop()
yield top
if (++emitted >= limit) return
} else break
}
if (esc.startsWith(RESERVED_START)) {
await reseek(prefix + RESERVED_END)
continue
}
if (cut === -1) {
// Own key: defer until we know no sibling extending it with a low char comes next.
pending.push({ esc, hasOwn: true, value: r.value[1] })
continue
}
// Descendant of a child with no own value (an own key would have sorted first).
yield { esc, hasOwn: false, value: undefined }
if (++emitted >= limit) return
await reseek(prefix + esc + succSep)
}
} finally {
await it.return?.()
}
}
async function* reverseScan(
driver: KVDriver,
prefix: string,
end: string | undefined,
sep: string,
opts: ChildScanOptions,
limit: number,
): AsyncGenerator<ChildHit> {
const sepCode = sep.charCodeAt(0)
const values = opts.values === true
const needOwn = values || opts.own === true
// Segments already emitted whose own key may still appear (it sorts after everything else of theirs).
const done: string[] = []
let emitted = 0
let lt = end
if (opts.after !== undefined) {
const escA = escapeSegment(opts.after, sep)
lt = prefix + escA + sep
// `after` itself and shorter siblings extending into it with a low char sort after it in
// child order but their own keys fall below `lt`: mark them done.
done.push(...lowPrefixes(escA, sepCode), escA)
}
const prefixLen = prefix.length
let it = driver.scan({ gte: prefix, lt, reverse: true, values })[Symbol.asyncIterator]()
const reseek = async (below: string) => {
await it.return?.()
it = driver.scan({ gte: prefix, lt: below, reverse: true, values })[Symbol.asyncIterator]()
}
try {
for (;;) {
const r = await it.next()
if (r.done) return
const key = r.value[0]
const cut = key.indexOf(sep, prefixLen)
const esc = cut === -1 ? key.slice(prefixLen) : key.slice(prefixLen, cut)
if (esc.startsWith(RESERVED_START)) {
await reseek(prefix + RESERVED_START)
continue
}
while (done.length && !esc.startsWith(done[done.length - 1]!)) done.pop()
if (done.length && done[done.length - 1] === esc) {
done.pop()
if (cut !== -1) await reseek(prefix + esc + sep)
continue
}
// New child. Shorter siblings that it extends with a low char come before it in reverse
// order; any with descendants were already emitted, so only own-key-only ones are unknown.
const cands = lowPrefixes(esc, sepCode).filter((g) => !done.includes(g))
const probeOwn = cut !== -1 && needOwn
let found: unknown[] = NONE
if (cands.length || probeOwn) {
const probe = cands.map((g) => prefix + g)
if (probeOwn) probe.push(prefix + esc)
found = await driver.getMany(probe)
}
for (let i = 0; i < cands.length; i++) {
if (found[i] === undefined) continue
yield { esc: cands[i]!, hasOwn: true, value: values ? found[i] : undefined }
done.push(cands[i]!)
if (++emitted >= limit) return
}
const own = cut === -1 ? r.value[1] : probeOwn ? found[cands.length] : undefined
yield { esc, hasOwn: cut === -1 || own !== undefined, value: values ? own : undefined }
done.push(esc)
if (++emitted >= limit) return
if (cut !== -1) await reseek(prefix + esc + sep)
}
} finally {
await it.return?.()
}
}
src/kv/driver.ts
-77
View File
@@ -1,77 +0,0 @@
export interface ScanOptions {
gte?: string
lt?: string
limit?: number
reverse?: boolean
/** When false/omitted, yielded values are `undefined`. */
values?: boolean
}
export type BatchOp = { type: 'set'; key: string; value: unknown } | { type: 'delete'; key: string }
/**
* A flat, string-keyed backend. Values are JSON-serialisable; byte-oriented backends serialise
* internally with a {@link Codec}. All path, index and typing logic lives above this interface.
*/
export interface KVDriver {
/** Same order as `keys`; `undefined` for missing keys. */
getMany(keys: string[]): Promise<(unknown | undefined)[]>
setMany(entries: [key: string, value: unknown][]): Promise<void>
deleteMany(keys: string[]): Promise<void>
/** MUST yield in key order (descending when `reverse`). `gte` is inclusive, `lt` exclusive. */
scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]>
// Optional capabilities. Stores feature-detect them and fall back where possible.
/** Writes each entry only if its key is absent. `true` = written. */
setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]>
/** Applies all ops atomically. */
batch?(ops: BatchOp[]): Promise<void>
deleteRange?(gte: string, lt: string): Promise<void>
}
/** Serialises values for byte/string-oriented backends. */
export interface Codec {
encode(value: unknown): string
decode(encoded: string): unknown
}
export const JsonCodec: Codec = {
encode: (value) => JSON.stringify(value),
decode: (encoded) => JSON.parse(encoded),
}
/**
* Base class for drivers: implement the four required methods and get `deleteRange`
* (scan + deleteMany in chunks) and single-key helpers for free.
*/
export abstract class BaseKVDriver implements KVDriver {
abstract getMany(keys: string[]): Promise<(unknown | undefined)[]>
abstract setMany(entries: [string, unknown][]): Promise<void>
abstract deleteMany(keys: string[]): Promise<void>
abstract scan(opts: ScanOptions): AsyncIterable<[string, unknown]>
/** Chunk size used by the default `deleteRange`. */
protected deleteRangeChunk = 1000
async deleteRange(gte: string, lt: string): Promise<void> {
for (;;) {
const keys: string[] = []
for await (const [key] of this.scan({ gte, lt, limit: this.deleteRangeChunk })) keys.push(key)
if (keys.length === 0) return
await this.deleteMany(keys)
if (keys.length < this.deleteRangeChunk) return
}
}
async get(key: string): Promise<unknown | undefined> {
return (await this.getMany([key]))[0]
}
set(key: string, value: unknown): Promise<void> {
return this.setMany([[key, value]])
}
delete(key: string): Promise<void> {
return this.deleteMany([key])
}
}
src/kv/index.ts
-6
View File
@@ -1,6 +0,0 @@
export * from './driver.js'
export * from './memory.js'
export * from './sorted.js'
export * from './node.js'
export * from './indexes.js'
export * from './children.js'
src/kv/indexes.ts
-157
View File
@@ -1,157 +0,0 @@
import { indexValueCodec, isIndexValue, type IndexValue, type IndexValueCodec } from '../core/encoding.js'
import { escapeSegment, RESERVED, succ } from '../core/path.js'
import { PatternTrie, patternsOverlap, type TrieState } from '../core/trie.js'
export type { IndexValue }
/** Maps a child's value to index value(s). An array means multi-entry; `undefined` means not indexed. */
export type IndexFn<V = any> = (value: V) => IndexValue | readonly IndexValue[] | undefined
/** Index name → function, for the direct children of nodes matching one pattern. */
export type IndexDefs = Record<string, IndexFn>
/** Path pattern (`*` = one segment, `''` = root) → index definitions. */
export type IndexDeclarations = Record<string, IndexDefs>
/** A value or a range over index values. Bounds combine (`{ gte, lt }`). */
export type IndexQuery = IndexValue | IndexRange
export interface IndexRange {
gt?: IndexValue
gte?: IndexValue
lt?: IndexValue
lte?: IndexValue
}
export function isIndexRange(q: unknown): q is IndexRange {
return typeof q === 'object' && q !== null
}
interface CompiledIndex {
name: string
/** `escapeSegment(name) + sep` */
head: string
fn: IndexFn
}
/** Indexes over the direct children of one concrete node. Entry keys are rooted at `base`. */
export class BoundIndexes {
readonly #byName: Map<string, CompiledIndex>
readonly #list: CompiledIndex[]
readonly #codec: IndexValueCodec
constructor(
defs: CompiledIndex[],
/** `<nodePrefix>~i<sep>` */
readonly base: string,
readonly sep: string,
) {
this.#list = defs
this.#byName = new Map(defs.map((d) => [d.name, d]))
this.#codec = indexValueCodec(sep)
}
has(name: string): boolean {
return this.#byName.has(name)
}
/** All index entry keys for a child (escaped segment) holding `value`. */
entriesFor(escChild: string, value: unknown, out: Set<string> = new Set()): Set<string> {
if (value === undefined) return out
for (const idx of this.#list) this.#entriesOf(idx, escChild, value, out)
return out
}
#entriesOf(idx: CompiledIndex, escChild: string, value: unknown, out: Set<string>): void {
const r = idx.fn(value)
if (r === undefined) return
const stem = this.base + idx.head
const tail = this.sep + escChild
if (Array.isArray(r)) {
for (const v of r as IndexValue[]) out.add(stem + this.#encode(idx.name, v) + tail)
} else out.add(stem + this.#encode(idx.name, r as IndexValue) + tail)
}
#encode(name: string, v: unknown): string {
if (!isIndexValue(v)) throw new TypeError(`Index "${name}" returned a non-indexable value: ${String(v)}`)
return this.#codec.encode(v)
}
/** Whether `entryKey` is among the entries `value` currently produces for index `name`. */
matches(name: string, escChild: string, value: unknown, entryKey: string): boolean {
if (value === undefined) return false
const out = new Set<string>()
this.#entriesOf(this.#get(name), escChild, value, out)
return out.has(entryKey)
}
/** Scan bounds over the entries of index `name` for a query. */
range(name: string, q: IndexQuery): { gte: string; lt: string } {
const stem = this.base + this.#get(name).head
const succSep = succ(this.sep)
const enc = (v: IndexValue) => stem + this.#encode(name, v)
if (!isIndexRange(q)) {
const k = enc(q)
return { gte: k + this.sep, lt: k + succSep }
}
let gte = stem
let lt = stem.slice(0, -1) + succSep
if (q.gte !== undefined) gte = maxStr(gte, enc(q.gte))
if (q.gt !== undefined) gte = maxStr(gte, enc(q.gt) + succSep)
if (q.lt !== undefined) lt = minStr(lt, enc(q.lt))
if (q.lte !== undefined) lt = minStr(lt, enc(q.lte) + succSep)
return { gte, lt }
}
/** Splits an entry key under index `name` into the escaped child segment. */
childOf(name: string, entryKey: string): string {
const start = this.base.length + this.#get(name).head.length
return entryKey.slice(entryKey.indexOf(this.sep, start) + 1)
}
#get(name: string): CompiledIndex {
const idx = this.#byName.get(name)
if (!idx) throw new TypeError(`Unknown index "${name}"`)
return idx
}
}
const maxStr = (a: string, b: string) => (a > b ? a : b)
const minStr = (a: string, b: string) => (a < b ? a : b)
/** Compiles index declarations into a pattern trie and binds them to nodes. */
export class IndexRegistry {
readonly trie: PatternTrie<IndexDefs>
readonly empty: boolean
constructor(
decls: IndexDeclarations,
readonly sep: string,
) {
this.trie = new PatternTrie(sep)
const patterns = Object.keys(decls)
for (let i = 0; i < patterns.length; i++) {
for (let j = i + 1; j < patterns.length; j++) {
const a = patterns[i]!
const b = patterns[j]!
if (!patternsOverlap(a, b, sep)) continue
const clash = Object.keys(decls[a]!).filter((n) => n in decls[b]!)
if (clash.length) throw new TypeError(`Index name(s) ${clash.join(', ')} declared on overlapping patterns "${a}" and "${b}"`)
}
}
for (const p of patterns) this.trie.add(p, decls[p]!)
this.empty = patterns.length === 0
}
/** Indexes over the children of the node with trie state `state` and key prefix `prefix`. */
bind(state: TrieState<IndexDefs>, prefix: string): BoundIndexes | undefined {
if (this.empty) return undefined
const found = PatternTrie.values(state)
if (found.length === 0) return undefined
const defs: CompiledIndex[] = []
for (const { value } of found) {
for (const [name, fn] of Object.entries(value)) defs.push({ name, head: escapeSegment(name, this.sep) + this.sep, fn })
}
return new BoundIndexes(defs, prefix + RESERVED.index + this.sep, this.sep)
}
}
src/kv/memory.ts
-93
View File
@@ -1,93 +0,0 @@
import { BaseKVDriver, type BatchOp, type ScanOptions } from './driver.js'
import { ChunkedSortedMap, type SortedMap } from './sorted.js'
export interface InMemoryKVDriverOptions {
/**
* Deep-copy values on write and read so callers can't mutate stored state through a shared
* reference (matching serialising drivers). Values must be JSON-compatible. Default `true`.
* Disable for speed when values are treated as immutable.
*/
clone?: boolean
}
/**
* In-memory driver over a sorted map. Implements every optional capability:
* `setIfAbsent`, atomic `batch` and `deleteRange`.
*/
export class InMemoryKVDriver extends BaseKVDriver {
readonly #data: SortedMap<unknown> = new ChunkedSortedMap()
readonly #clone: (v: unknown) => unknown
constructor(opts: InMemoryKVDriverOptions = {}) {
super()
this.#clone = opts.clone === false ? identity : cloneValue
}
get size(): number {
return this.#data.size
}
async getMany(keys: string[]): Promise<unknown[]> {
const out = new Array<unknown>(keys.length)
for (let i = 0; i < keys.length; i++) out[i] = this.#clone(this.#data.get(keys[i]!))
return out
}
async setMany(entries: [string, unknown][]): Promise<void> {
const clone = this.#clone
this.#data.setMany(entries.map(([k, v]) => [k, clone(v)]))
}
async deleteMany(keys: string[]): Promise<void> {
for (const key of keys) this.#data.delete(key)
}
async setIfAbsent(entries: [string, unknown][]): Promise<boolean[]> {
return entries.map(([key, value]) => {
if (this.#data.has(key)) return false
this.#data.set(key, this.#clone(value))
return true
})
}
async batch(ops: BatchOp[]): Promise<void> {
// Applied synchronously within one microtask, so no other operation can observe a partial batch.
for (const op of ops) {
if (op.type === 'set') this.#data.set(op.key, this.#clone(op.value))
else this.#data.delete(op.key)
}
}
override async deleteRange(gte: string, lt: string): Promise<void> {
const doomed = [...this.#data.keys({ gte, lt })]
for (const key of doomed) this.#data.delete(key)
}
async *scan(opts: ScanOptions): AsyncGenerator<[string, unknown]> {
const { gte, lt, reverse = false, values = false } = opts
let remaining = opts.limit ?? Infinity
if (remaining <= 0) return
// The key iterator re-seeks by itself if the map changes while we're suspended.
for (const key of this.#data.keys({ gte, lt, reverse })) {
yield [key, values ? this.#clone(this.#data.get(key)) : undefined]
if (--remaining <= 0) return
}
}
}
function identity(v: unknown): unknown {
return v
}
/** Deep copy of a JSON-compatible value (much faster than `structuredClone`). */
function cloneValue(v: unknown): unknown {
if (v === null || typeof v !== 'object') return v
if (Array.isArray(v)) {
const out = new Array<unknown>(v.length)
for (let i = 0; i < v.length; i++) out[i] = cloneValue(v[i])
return out
}
const out: Record<string, unknown> = {}
for (const k in v) out[k] = cloneValue((v as Record<string, unknown>)[k])
return out
}
src/kv/node.ts
-402
View File
@@ -1,402 +0,0 @@
import { DEFAULT_SEPARATOR, escapeSegment, succ, toSegment, unescapeSegment, validateSeparator } from '../core/path.js'
import type { TrieState } from '../core/trie.js'
import type { AnySpec, ChildOf, EntryOf, KeyOf, KeyString, Spec, StringEntryOf, ValueOf } from '../types/schema.js'
import { scanChildren } from './children.js'
import type { BatchOp, KVDriver } from './driver.js'
import { BoundIndexes, IndexRegistry, isIndexRange, type IndexDeclarations, type IndexDefs, type IndexQuery } from './indexes.js'
export interface KVStorageOptions {
/** Single printable ASCII character other than `%` and `~`. Default `/`. */
separator?: string
/** Path pattern → index definitions over that node's direct children. */
indexes?: IndexDeclarations
/**
* Delete stale index entries found by `findBy` (read-repair). Default `true`.
* See DESIGN.md §12 (M4) for the race with concurrent non-atomic writers.
*/
readRepair?: boolean
}
export interface KeysOptions {
limit?: number
/** Start after this child key (exclusive), in the direction of travel. */
after?: string | number
reverse?: boolean
}
export interface TreeOptions {
/** Levels below this node to include; `1` = direct children only. */
depth: number
}
export interface FindByOptions {
limit?: number
reverse?: boolean
}
/** @internal */
export interface KVContext {
readonly driver: KVDriver
readonly sep: string
readonly registry: IndexRegistry
readonly readRepair: boolean
}
interface Slot {
key: string
esc: string
value: unknown
}
const FIND_CHUNK = 128
const DELETE_CHUNK = 1000
/**
* A store over its direct children. `child(key)` re-roots to that key (sync, no I/O).
* Methods take an optional leading key: without it they act on this node's own value,
* with it on that child's. The choice is made by argument count; a passed-but-`undefined`
* key throws.
*/
export class KVNode<S extends Spec = AnySpec> {
/** Unescaped path segments. */
readonly path: readonly string[]
/** This node's own key (escaped). `''` for the root. */
readonly selfKey: string
/** Prefix of every descendant key. `''` for the root. */
readonly prefix: string
/** Exclusive upper bound of the subtree range; `undefined` for the root. */
readonly end: string | undefined
readonly #ctx: KVContext
readonly #esc: string
readonly #state: TrieState<IndexDefs>
readonly #selfIdx: BoundIndexes | undefined
readonly #childIdx: BoundIndexes | undefined
/** @internal Use `KVStorage` and `child()`. */
constructor(ctx: KVContext, path: readonly string[], selfKey: string, esc: string, state: TrieState<IndexDefs>, selfIdx: BoundIndexes | undefined) {
this.#ctx = ctx
this.path = path
this.selfKey = selfKey
this.#esc = esc
const root = path.length === 0
this.prefix = root ? '' : selfKey + ctx.sep
this.end = root ? undefined : selfKey + succ(ctx.sep)
this.#state = state
this.#selfIdx = selfIdx
this.#childIdx = ctx.registry.bind(state, this.prefix)
}
get driver(): KVDriver {
return this.#ctx.driver
}
get separator(): string {
return this.#ctx.sep
}
get isRoot(): boolean {
return this.path.length === 0
}
child<K extends KeyOf<S>>(key: K): KVNode<ChildOf<S, K>> {
const seg = toSegment(key)
const esc = escapeSegment(seg, this.#ctx.sep)
return new KVNode(this.#ctx, [...this.path, seg], this.prefix + esc, esc, this.#ctx.registry.trie.step(this.#state, seg), this.#childIdx)
}
/** Re-types this node. No runtime effect. */
as<S2 extends Spec>(): KVNode<S2> {
return this as unknown as KVNode<S2>
}
get(): Promise<ValueOf<S> | undefined>
get<K extends KeyOf<S>>(key: K): Promise<ValueOf<ChildOf<S, K>> | undefined>
async get(...args: unknown[]): Promise<unknown> {
const t = this.#resolve(args, 0)
return (await this.#ctx.driver.getMany([t.key]))[0]
}
async getMany<K extends KeyOf<S>>(keys: readonly K[]): Promise<(ValueOf<ChildOf<S, K>> | undefined)[]> {
if (keys.length === 0) return []
return (await this.#ctx.driver.getMany(keys.map((k) => this.#childKey(k)))) as (ValueOf<ChildOf<S, K>> | undefined)[]
}
set(value: ValueOf<S>): Promise<void>
set<K extends KeyOf<S>>(key: K, value: ValueOf<ChildOf<S, K>>): Promise<void>
async set(...args: unknown[]): Promise<void> {
const t = this.#resolve(args, 1)
const value = args[args.length - 1]
if (value === undefined) throw new TypeError('Cannot set undefined; use delete()')
return this.#write(t.idx, [{ key: t.key, esc: t.esc, value }])
}
async setMany(entries: readonly EntryOf<S>[]): Promise<void> {
if (entries.length === 0) return
const slots = entries.map(([k, value]): Slot => {
if (value === undefined) throw new TypeError('Cannot set undefined; use delete()')
const esc = escapeSegment(toSegment(k), this.#ctx.sep)
return { key: this.prefix + esc, esc, value }
})
return this.#write(this.#childIdx, slots)
}
/** Deletes the own value (or a child's own value) only, not descendants. */
delete(): Promise<void>
delete<K extends KeyOf<S>>(key: K): Promise<void>
async delete(...args: unknown[]): Promise<void> {
const t = this.#resolve(args, 0)
return this.#write(t.idx, [{ key: t.key, esc: t.esc, value: undefined }])
}
async deleteMany(keys: readonly KeyOf<S>[]): Promise<void> {
if (keys.length === 0) return
const slots = keys.map((k): Slot => {
const esc = escapeSegment(toSegment(k), this.#ctx.sep)
return { key: this.prefix + esc, esc, value: undefined }
})
return this.#write(this.#childIdx, slots)
}
/** Deletes the own value and all descendants (or a child's). On the root, deletes everything. */
deleteTree(): Promise<void>
deleteTree<K extends KeyOf<S>>(key: K): Promise<void>
async deleteTree(...args: unknown[]): Promise<void> {
if (args.length === 0 && this.isRoot) return this.#deleteRange('', undefined)
const t = this.#resolve(args, 0)
await this.#write(t.idx, [{ key: t.key, esc: t.esc, value: undefined }])
await this.#deleteRange(t.key + this.#ctx.sep, t.key + succ(this.#ctx.sep))
}
/**
* Direct child keys (unescaped), including children that only have descendants.
* Order: see `scanChildren` — lexicographic by escaped segment followed by the separator.
*/
async *keys(opts: KeysOptions = {}): AsyncGenerator<KeyString<S>> {
const after = opts.after === undefined ? undefined : toSegment(opts.after)
for await (const hit of scanChildren(this.#ctx.driver, this.prefix, this.end, this.#ctx.sep, {
after,
reverse: opts.reverse,
limit: opts.limit,
})) {
yield unescapeSegment(hit.esc) as KeyString<S>
}
}
/** Direct children that have an own value, as `[key, value]`. */
async *entries(opts: KeysOptions = {}): AsyncGenerator<StringEntryOf<S>> {
const after = opts.after === undefined ? undefined : toSegment(opts.after)
let remaining = opts.limit ?? Infinity
if (remaining <= 0) return
for await (const hit of scanChildren(this.#ctx.driver, this.prefix, this.end, this.#ctx.sep, {
after,
reverse: opts.reverse,
values: true,
})) {
if (!hit.hasOwn) continue
yield [unescapeSegment(hit.esc), hit.value] as StringEntryOf<S>
if (--remaining <= 0) return
}
}
/**
* Assembles descendants up to `depth` levels into a plain object: leaves become values,
* nodes with children become objects. A node with both an own value and children exposes the
* own value under the `''` property (empty keys are invalid, so it can't collide).
* Reserved (`~…`) subtrees are skipped. This node's own value is not included.
*/
async tree<T = Record<string, unknown>>(opts: TreeOptions): Promise<T> {
const { depth } = opts
if (!(depth >= 1)) throw new RangeError('tree() depth must be >= 1')
const { driver, sep } = this.#ctx
type TNode = { own?: unknown; has: boolean; kids?: Map<string, TNode> }
const root: TNode = { has: false }
const prefix = this.prefix
let it = driver.scan({ gte: prefix, lt: this.end, values: true })[Symbol.asyncIterator]()
const reseek = async (gte: string) => {
await it.return?.()
it = driver.scan({ gte, lt: this.end, values: true })[Symbol.asyncIterator]()
}
try {
outer: for (;;) {
const r = await it.next()
if (r.done) break
const [key, value] = r.value
const segs = key.slice(prefix.length).split(sep)
let base = prefix
for (let i = 0; i < segs.length && i < depth; i++) {
if (segs[i]!.startsWith('~')) {
await reseek(base + succ('~'))
continue outer
}
base += segs[i]! + sep
}
if (segs.length > depth) {
await reseek(prefix + segs.slice(0, depth).join(sep) + succ(sep))
continue
}
let node = root
for (const s of segs) {
const kids = (node.kids ??= new Map())
const seg = unescapeSegment(s)
let next = kids.get(seg)
if (!next) kids.set(seg, (next = { has: false }))
node = next
}
node.own = value
node.has = true
}
} finally {
await it.return?.()
}
const build = (n: TNode): unknown => {
if (!n.kids) return n.own
const obj: Record<string, unknown> = {}
if (n.has) obj[''] = n.own
for (const [k, v] of n.kids) obj[k] = build(v)
return obj
}
return (root.kids ? build(root) : {}) as T
}
/**
* Looks up direct children by an index declared for this node, by exact value or range.
* Each hit's value is re-read and re-checked; stale entries are skipped (and deleted when
* `readRepair` is on). Multi-entry hits are de-duplicated.
*/
async *findBy(index: string, query: IndexQuery, opts: FindByOptions = {}): AsyncGenerator<StringEntryOf<S>> {
const idx = this.#childIdx
if (!idx || !idx.has(index)) throw new TypeError(`No index "${index}" on node "${this.selfKey}"`)
const { driver, readRepair } = this.#ctx
const { gte, lt } = idx.range(index, query)
let remaining = opts.limit ?? Infinity
if (remaining <= 0) return
const seen = isIndexRange(query) ? new Set<string>() : undefined
let chunk: string[] = []
const it = driver.scan({ gte, lt, reverse: opts.reverse })[Symbol.asyncIterator]()
// Processes one chunk of entry keys; returns false when the limit is reached.
const flush = async function* (this: KVNode<S>): AsyncGenerator<StringEntryOf<S>, boolean> {
const entries = chunk
chunk = []
const escs = entries.map((e) => idx.childOf(index, e))
const values = await driver.getMany(escs.map((e) => this.prefix + e))
const stale: string[] = []
try {
for (let i = 0; i < entries.length; i++) {
const esc = escs[i]!
if (!idx.matches(index, esc, values[i], entries[i]!)) {
stale.push(entries[i]!)
continue
}
if (seen) {
if (seen.has(esc)) continue
seen.add(esc)
}
yield [unescapeSegment(esc), values[i]] as StringEntryOf<S>
if (--remaining <= 0) return false
}
return true
} finally {
if (readRepair && stale.length) await driver.deleteMany(stale)
}
}
try {
for (;;) {
const r = await it.next()
if (r.done) break
chunk.push(r.value[0])
// Small first chunk so `findOne`-style lookups cost one probe.
if (chunk.length >= (remaining < FIND_CHUNK ? remaining : FIND_CHUNK)) {
if (!(yield* flush.call(this))) return
}
}
if (chunk.length) yield* flush.call(this)
} finally {
await it.return?.()
}
}
// ---------------------------------------------------------------- internals
#childKey(key: unknown): string {
return this.prefix + escapeSegment(toSegment(key), this.#ctx.sep)
}
/** Self vs child by argument count; `selfArity` is the self form's argument count. */
#resolve(args: readonly unknown[], selfArity: number): { key: string; esc: string; idx: BoundIndexes | undefined } {
if (args.length <= selfArity) {
if (this.isRoot) throw new TypeError('The root node has no own value')
return { key: this.selfKey, esc: this.#esc, idx: this.#selfIdx }
}
const esc = escapeSegment(toSegment(args[0]), this.#ctx.sep)
return { key: this.prefix + esc, esc, idx: this.#childIdx }
}
/** Writes slots (`value: undefined` = delete), maintaining indexes when present. */
async #write(idx: BoundIndexes | undefined, slots: Slot[]): Promise<void> {
const { driver } = this.#ctx
if (!idx) {
const sets: [string, unknown][] = []
const dels: string[] = []
for (const s of slots) {
if (s.value === undefined) dels.push(s.key)
else sets.push([s.key, s.value])
}
if (sets.length) await driver.setMany(sets)
if (dels.length) await driver.deleteMany(dels)
return
}
// Last write per key wins.
const byKey = new Map<string, Slot>()
for (const s of slots) byKey.set(s.key, s)
const unique = [...byKey.values()]
const old = await driver.getMany(unique.map((s) => s.key))
const adds: string[] = []
const removes: string[] = []
for (let i = 0; i < unique.length; i++) {
const s = unique[i]!
const before = idx.entriesFor(s.esc, old[i])
const after = idx.entriesFor(s.esc, s.value)
for (const e of after) if (!before.has(e)) adds.push(e)
for (const e of before) if (!after.has(e)) removes.push(e)
}
if (driver.batch) {
const ops: BatchOp[] = []
for (const k of adds) ops.push({ type: 'set', key: k, value: true })
for (const s of unique) ops.push(s.value === undefined ? { type: 'delete', key: s.key } : { type: 'set', key: s.key, value: s.value })
for (const k of removes) ops.push({ type: 'delete', key: k })
return driver.batch(ops)
}
// Non-atomic: adds first, then values, then removes. Entries are hints; findBy re-checks.
if (adds.length) await driver.setMany(adds.map((k) => [k, true]))
const sets = unique.filter((s) => s.value !== undefined)
const dels = unique.filter((s) => s.value === undefined)
if (sets.length) await driver.setMany(sets.map((s) => [s.key, s.value]))
if (dels.length) await driver.deleteMany(dels.map((s) => s.key))
if (removes.length) await driver.deleteMany(removes)
}
async #deleteRange(gte: string, lt: string | undefined): Promise<void> {
const { driver } = this.#ctx
if (lt !== undefined && driver.deleteRange) return driver.deleteRange(gte, lt)
for (;;) {
const keys: string[] = []
for await (const [k] of driver.scan({ gte, lt, limit: DELETE_CHUNK })) keys.push(k)
if (keys.length === 0) return
await driver.deleteMany(keys)
if (keys.length < DELETE_CHUNK) return
}
}
}
/** The root `KVNode` over a driver. */
export class KVStorage<S extends Spec = AnySpec> extends KVNode<S> {
constructor(driver: KVDriver, opts: KVStorageOptions = {}) {
const sep = opts.separator ?? DEFAULT_SEPARATOR
validateSeparator(sep)
const registry = new IndexRegistry(opts.indexes ?? {}, sep)
super({ driver, sep, registry, readRepair: opts.readRepair ?? true }, [], '', '', registry.trie.rootState, undefined)
}
}
src/kv/sorted.ts
-181
View File
@@ -1,181 +0,0 @@
/**
* Ordered string-keyed map used by the in-memory driver. Kept behind an interface so the
* implementation can be swapped (e.g. for a B-tree) without touching the driver.
*/
export interface SortedMap<V> {
readonly size: number
get(key: string): V | undefined
has(key: string): boolean
set(key: string, value: V): void
/** Inserts many entries; may be faster than repeated `set`. */
setMany(entries: Iterable<[string, V]>): void
delete(key: string): boolean
/**
* Keys in `[gte, lt)` in order (descending when `reverse`). The iterator stays valid across
* mutations between steps: it resumes after (before) the last key it returned.
*/
keys(range: { gte?: string; lt?: string; reverse?: boolean }): IterableIterator<string>
}
/**
* Sorted chunks of at most `maxChunk` keys (split in half on overflow) plus a `Map` for values.
* Inserts and deletes cost O(log n + maxChunk); iteration is O(1) per step while the map
* is unchanged and re-seeks in O(log n) after a mutation.
*/
export class ChunkedSortedMap<V> implements SortedMap<V> {
constructor(readonly maxChunk = 1024) {}
#chunks: string[][] = [[]]
readonly #values = new Map<string, V>()
#version = 0
get size(): number {
return this.#values.size
}
get(key: string): V | undefined {
return this.#values.get(key)
}
has(key: string): boolean {
return this.#values.has(key)
}
set(key: string, value: V): void {
if (!this.#values.has(key)) this.#insert(key)
this.#values.set(key, value)
}
setMany(entries: Iterable<[string, V]>): void {
const fresh: string[] = []
for (const [key, value] of entries) {
if (!this.#values.has(key)) fresh.push(key)
this.#values.set(key, value)
}
if (fresh.length === 0) return
if (fresh.length < 64 || fresh.length * 8 < this.#values.size) {
for (const key of fresh) this.#insert(key)
return
}
// Bulk path: merge and re-chunk, O(n + k log k).
fresh.sort(cmp)
const old = this.#chunks.flat()
const merged = new Array<string>(old.length + fresh.length)
let i = 0
let j = 0
let k = 0
while (i < old.length && j < fresh.length) merged[k++] = old[i]! < fresh[j]! ? old[i++]! : fresh[j++]!
while (i < old.length) merged[k++] = old[i++]!
while (j < fresh.length) merged[k++] = fresh[j++]!
const half = Math.max(1, this.maxChunk >>> 1)
this.#chunks = []
for (let s = 0; s < merged.length; s += half) this.#chunks.push(merged.slice(s, s + half))
this.#version++
}
delete(key: string): boolean {
if (!this.#values.delete(key)) return false
const c = this.#chunkFor(key)
const chunk = this.#chunks[c]!
chunk.splice(lowerBound(chunk, key), 1)
if (chunk.length === 0 && this.#chunks.length > 1) this.#chunks.splice(c, 1)
this.#version++
return true
}
*keys(range: { gte?: string; lt?: string; reverse?: boolean }): IterableIterator<string> {
const { gte, lt, reverse = false } = range
let version = this.#version
let c: number
let i: number
// Position of the first candidate.
if (reverse) {
;[c, i] = lt === undefined ? this.#end() : this.#seek(lt, false)
;[c, i] = this.#prev(c, i)
} else {
;[c, i] = gte === undefined ? [0, 0] : this.#seek(gte, false)
;[c, i] = this.#normalize(c, i)
}
for (;;) {
const key = this.#chunks[c]?.[i]
if (key === undefined || c < 0) return
if (reverse ? gte !== undefined && key < gte : lt !== undefined && key >= lt) return
yield key
if (this.#version !== version) {
version = this.#version
;[c, i] = this.#seek(key, !reverse)
if (reverse) [c, i] = this.#prev(c, i)
else [c, i] = this.#normalize(c, i)
} else if (reverse) [c, i] = this.#prev(c, i)
else [c, i] = this.#normalize(c, i + 1)
}
}
#insert(key: string): void {
const c = this.#chunkFor(key)
const chunk = this.#chunks[c]!
chunk.splice(lowerBound(chunk, key), 0, key)
if (chunk.length > this.maxChunk) this.#chunks.splice(c + 1, 0, chunk.splice(this.maxChunk >>> 1))
this.#version++
}
/** Index of the last chunk whose first key is <= key (0 if none). */
#chunkFor(key: string): number {
const chunks = this.#chunks
let lo = 0
let hi = chunks.length - 1
while (lo < hi) {
const mid = (lo + hi + 1) >>> 1
if (chunks[mid]![0]! <= key) lo = mid
else hi = mid - 1
}
return lo
}
/** Position of the first key `>= key` (`> key` when exclusive); may point one past a chunk's end. */
#seek(key: string, exclusive: boolean): [number, number] {
const c = this.#chunkFor(key)
const chunk = this.#chunks[c]!
let i = lowerBound(chunk, key)
if (exclusive && chunk[i] === key) i++
return [c, i]
}
#end(): [number, number] {
const c = this.#chunks.length - 1
return [c, this.#chunks[c]!.length]
}
/** Moves a position that is past its chunk's end to the start of the next chunk. */
#normalize(c: number, i: number): [number, number] {
while (c < this.#chunks.length && i >= this.#chunks[c]!.length) {
c++
i = 0
}
return [c, i]
}
#prev(c: number, i: number): [number, number] {
i--
while (i < 0 && c > 0) {
c--
i = this.#chunks[c]!.length - 1
}
return i < 0 ? [-1, -1] : [c, i]
}
}
function lowerBound(arr: string[], key: string): number {
let lo = 0
let hi = arr.length
while (lo < hi) {
const mid = (lo + hi) >>> 1
if (arr[mid]! < key) lo = mid + 1
else hi = mid
}
return lo
}
function cmp(a: string, b: string): number {
return a < b ? -1 : a > b ? 1 : 0
}
src/testing/index.ts
-1
View File
@@ -1 +0,0 @@
export * from './kv-conformance.js'
src/testing/kv-conformance.ts
-171
View File
@@ -1,171 +0,0 @@
import { beforeEach, describe, expect, it } from 'vitest'
import type { KVDriver, ScanOptions } from '../kv/driver.js'
/**
* Reusable conformance suite for {@link KVDriver} implementations. Call it from a Vitest test file:
*
* ```ts
* runKVDriverConformance('MyDriver', () => new MyDriver())
* ```
*
* `makeDriver` must return an empty driver each time.
*/
export function runKVDriverConformance(name: string, makeDriver: () => KVDriver | Promise<KVDriver>): void {
describe(`KVDriver conformance: ${name}`, () => {
let d: KVDriver
beforeEach(async () => {
d = await makeDriver()
})
async function collect(opts: ScanOptions): Promise<[string, unknown][]> {
const out: [string, unknown][] = []
for await (const e of d.scan(opts)) out.push(e)
return out
}
const keysOf = async (opts: ScanOptions) => (await collect(opts)).map(([k]) => k)
async function seed(): Promise<void> {
// Deliberately inserted out of order.
await d.setMany([
['b', 2],
['a/2', { n: 'a2' }],
['c', [3]],
['a', 1],
['a/1', 'a1'],
['a0', null],
['10', 10],
['9', 9],
])
}
it('round-trips JSON values with getMany in key order, undefined for missing', async () => {
await seed()
expect(await d.getMany(['c', 'missing', 'a/2', 'a0', 'a'])).toEqual([[3], undefined, { n: 'a2' }, null, 1])
expect(await d.getMany([])).toEqual([])
})
it('overwrites on setMany and deletes with deleteMany', async () => {
await seed()
await d.setMany([['a', 'one']])
await d.deleteMany(['b', 'missing'])
expect(await d.getMany(['a', 'b'])).toEqual(['one', undefined])
})
it('scans all keys in lexicographic order', async () => {
await seed()
expect(await keysOf({})).toEqual(['10', '9', 'a', 'a/1', 'a/2', 'a0', 'b', 'c'])
})
it('honours gte (inclusive) and lt (exclusive)', async () => {
await seed()
expect(await keysOf({ gte: 'a/', lt: 'a0' })).toEqual(['a/1', 'a/2'])
expect(await keysOf({ gte: 'a', lt: 'b' })).toEqual(['a', 'a/1', 'a/2', 'a0'])
expect(await keysOf({ gte: 'a0' })).toEqual(['a0', 'b', 'c'])
expect(await keysOf({ lt: 'a' })).toEqual(['10', '9'])
expect(await keysOf({ gte: 'zz' })).toEqual([])
expect(await keysOf({ gte: 'b', lt: 'b' })).toEqual([])
})
it('scans in reverse within bounds', async () => {
await seed()
expect(await keysOf({ reverse: true })).toEqual(['c', 'b', 'a0', 'a/2', 'a/1', 'a', '9', '10'])
expect(await keysOf({ reverse: true, gte: 'a', lt: 'a0' })).toEqual(['a/2', 'a/1', 'a'])
})
it('honours limit, forwards and in reverse', async () => {
await seed()
expect(await keysOf({ limit: 2 })).toEqual(['10', '9'])
expect(await keysOf({ limit: 2, reverse: true, lt: 'b' })).toEqual(['a0', 'a/2'])
expect(await keysOf({ limit: 0 })).toEqual([])
})
it('yields values only when values: true', async () => {
await seed()
expect(await collect({ gte: 'a/', lt: 'a0' })).toEqual([
['a/1', undefined],
['a/2', undefined],
])
expect(await collect({ gte: 'a/', lt: 'a0', values: true })).toEqual([
['a/1', 'a1'],
['a/2', { n: 'a2' }],
])
})
it('tolerates writes while a scan is suspended', async () => {
await d.setMany([
['k1', 1],
['k2', 2],
['k3', 3],
])
const seen: string[] = []
for await (const [key] of d.scan({})) {
seen.push(key)
if (key === 'k1') await d.deleteMany(['k1'])
}
expect(seen).toEqual(['k1', 'k2', 'k3'])
})
it('does not let callers mutate stored values through returned references', async () => {
await d.setMany([['o', { n: 1 }]])
const [v] = (await d.getMany(['o'])) as [{ n: number }]
v.n = 2
expect(await d.getMany(['o'])).toEqual([{ n: 1 }])
})
it('stores unicode keys and orders them by code unit', async () => {
await d.setMany([
['é', 1],
['z', 2],
['日本', 3],
])
expect(await keysOf({})).toEqual(['z', 'é', '日本'])
})
describe('optional: setIfAbsent', () => {
it('writes only absent keys and reports which were written', async (ctx) => {
if (!d.setIfAbsent) return ctx.skip()
await d.setMany([['x', 1]])
expect(
await d.setIfAbsent([
['x', 2],
['y', 3],
]),
).toEqual([false, true])
expect(await d.getMany(['x', 'y'])).toEqual([1, 3])
})
it('only the first of concurrent claims wins', async (ctx) => {
if (!d.setIfAbsent) return ctx.skip()
const results = await Promise.all([1, 2, 3, 4].map((n) => d.setIfAbsent!([['claim', n]])))
expect(results.flat().filter(Boolean)).toHaveLength(1)
})
})
describe('optional: batch', () => {
it('applies sets and deletes in order', async (ctx) => {
if (!d.batch) return ctx.skip()
await d.setMany([['gone', 1]])
await d.batch([
{ type: 'set', key: 'p', value: 1 },
{ type: 'delete', key: 'gone' },
{ type: 'set', key: 'q', value: { a: [1] } },
{ type: 'set', key: 'p', value: 2 },
])
expect(await d.getMany(['p', 'q', 'gone'])).toEqual([2, { a: [1] }, undefined])
})
})
describe('optional: deleteRange', () => {
it('deletes exactly [gte, lt)', async (ctx) => {
if (!d.deleteRange) return ctx.skip()
await seed()
await d.deleteRange('a/', 'a0')
expect(await keysOf({})).toEqual(['10', '9', 'a', 'a0', 'b', 'c'])
await d.deleteRange('a', 'b')
expect(await keysOf({})).toEqual(['10', '9', 'b', 'c'])
})
})
})
}
src/utils/document-path.ts
+32
View File
@@ -0,0 +1,32 @@
import { Json } from './json.js'
export type Doc = Record<string, unknown>
/** Dot-path access into plain JSON documents (`'address.city'`, `'tags.0'`). */
export class DocumentPath {
static get(doc: unknown, path: string): unknown {
let value = doc
for (const segment of path.split('.')) {
if (Array.isArray(value) || Json.isObject(value)) value = (value as Record<string, unknown>)[segment]
else return undefined
}
return value
}
/** Sets a path in place, creating intermediate objects. */
static set(doc: Doc, path: string, value: unknown): void {
const segments = path.split('.')
let target = doc as Record<string, unknown>
for (const segment of segments.slice(0, -1)) {
if (typeof target[segment] !== 'object' || target[segment] === null) target[segment] = {}
target = target[segment] as Record<string, unknown>
}
target[segments.at(-1)!] = value
}
static unset(doc: Doc, path: string): void {
const segments = path.split('.')
const parent = segments.length === 1 ? doc : DocumentPath.get(doc, segments.slice(0, -1).join('.'))
if (Json.isObject(parent)) delete parent[segments.at(-1)!]
}
}
src/utils/document-query.ts
+106
View File
@@ -0,0 +1,106 @@
import { DocumentPath, type Doc } from './document-path.js'
import { Json } from './json.js'
export type Comparison = { $eq?: unknown; $gt?: unknown; $gte?: unknown; $lt?: unknown; $lte?: unknown; $in?: unknown[] }
/** Field (dot path) → value (equality) or comparison operators. All fields must match. */
export type Filter = Record<string, unknown>
export interface FindOptions {
/** Field → 1 (ascending) or -1 (descending). */
sort?: Record<string, 1 | -1>
limit?: number
skip?: number
}
export interface FindResult<T = Doc> {
key: string
data: T
}
const OPERATORS = new Set(['$eq', '$gt', '$gte', '$lt', '$lte', '$in'])
/**
* Filter evaluation, sorting and paging for document stores that don't query natively.
*
* - `{ field: value }` matches equal values; `null` also matches a missing field.
* - Comparisons only match values of the same type (`$gt: 5` never matches a string).
* - On an array field, a condition matches if the array itself or any one element satisfies it.
*/
export class DocumentQuery {
constructor(
readonly filter: Filter = {},
readonly options: FindOptions = {},
) {}
matches(doc: Doc): boolean {
return Object.entries(this.filter).every(([path, condition]) => DocumentQuery.matchesValue(DocumentPath.get(doc, path), condition))
}
/** Whether one field value satisfies a condition. */
static matchesValue(value: unknown, condition: unknown): boolean {
const candidates = Array.isArray(value) ? [value, ...value] : [value]
if (!DocumentQuery.isComparison(condition)) return candidates.some((c) => DocumentQuery.equal(c, condition))
const ops = Object.entries(condition)
return candidates.some((c) => ops.every(([op, arg]) => DocumentQuery.compare(op, c, arg)))
}
/** Applies filter, sort, skip and limit to a stream of candidates. */
async *run(source: AsyncIterable<FindResult>): AsyncGenerator<FindResult> {
const { sort, skip = 0, limit = Infinity } = this.options
if (limit <= 0) return
if (sort && Object.keys(sort).length) {
const all: FindResult[] = []
for await (const result of source) if (this.matches(result.data)) all.push(result)
all.sort((a, b) => this.compareResults(a, b))
yield* all.slice(skip, skip + limit)
return
}
let skipped = 0
let emitted = 0
for await (const result of source) {
if (!this.matches(result.data)) continue
if (skipped++ < skip) continue
yield result
if (++emitted >= limit) return
}
}
private compareResults(a: FindResult, b: FindResult): number {
for (const [path, direction] of Object.entries(this.options.sort!)) {
const order = DocumentQuery.order(DocumentPath.get(a.data, path), DocumentPath.get(b.data, path))
if (order !== 0) return order * direction
}
return a.key < b.key ? -1 : a.key > b.key ? 1 : 0
}
private static isComparison(condition: unknown): condition is Comparison {
if (!Json.isObject(condition)) return false
const keys = Object.keys(condition)
if (keys.length === 0 || !keys.every((k) => k.startsWith('$'))) return false
for (const k of keys) if (!OPERATORS.has(k)) throw new TypeError(`Unsupported operator ${k}`)
return true
}
private static equal(value: unknown, expected: unknown): boolean {
return expected === null ? value === null || value === undefined : Json.equal(value, expected)
}
private static compare(op: string, value: unknown, arg: unknown): boolean {
if (op === '$eq') return DocumentQuery.equal(value, arg)
if (op === '$in') return (arg as unknown[]).some((x) => DocumentQuery.equal(value, x))
if (typeof value !== typeof arg || !['number', 'string', 'boolean'].includes(typeof value)) return false
const order = DocumentQuery.order(value, arg)
return op === '$gt' ? order > 0 : op === '$gte' ? order >= 0 : op === '$lt' ? order < 0 : order <= 0
}
/** Total order: missing/null < booleans < numbers < strings < anything else. */
static order(a: unknown, b: unknown): number {
const rank = (v: unknown) => (v === undefined || v === null ? 0 : (({ boolean: 1, number: 2, string: 3 } as Record<string, number>)[typeof v] ?? 4))
const ra = rank(a)
const rb = rank(b)
if (ra !== rb) return ra - rb
if (ra === 4) [a, b] = [JSON.stringify(a), JSON.stringify(b)]
return a === b ? 0 : (a as number) < (b as number) ? -1 : 1
}
}
src/utils/document-update.ts
+75
View File
@@ -0,0 +1,75 @@
import { DocumentPath, type Doc } from './document-path.js'
import { DocumentQuery } from './document-query.js'
import { Json } from './json.js'
/** Update operators. Paths may use dots. `$push`/`$addToSet` accept `{ $each: [...] }`. */
export interface UpdateSpec {
$set?: Record<string, unknown>
$unset?: Record<string, unknown>
$inc?: Record<string, number>
$push?: Record<string, unknown>
$pull?: Record<string, unknown>
$addToSet?: Record<string, unknown>
}
/**
* A resolved change to one document. Operators are resolved to final values (`$inc` becomes
* `set: { n: 6 }`), so replaying changes is deterministic.
*/
export type DocChange = { op: 'set'; doc: Doc } | { op: 'patch'; set: Record<string, unknown>; unset: string[] } | { op: 'delete' }
export class DocumentUpdate {
/** Resolves update operators against the current state into a patch. */
static resolve(state: Doc, spec: UpdateSpec): DocChange {
const set: Record<string, unknown> = {}
const unset: string[] = []
const array = (path: string) => {
const current = DocumentPath.get(state, path) ?? []
if (!Array.isArray(current)) throw new TypeError(`"${path}" is not an array`)
return Json.clone(current)
}
const items = (value: unknown) => (Json.isObject(value) && Array.isArray(value.$each) ? value.$each : [value])
for (const [op, fields] of Object.entries(spec)) {
for (const [path, value] of Object.entries(fields as Record<string, unknown>)) {
if (path in set || unset.includes(path)) throw new TypeError(`"${path}" is updated twice`)
switch (op) {
case '$set':
set[path] = Json.clone(value)
break
case '$unset':
unset.push(path)
break
case '$inc': {
const current = DocumentPath.get(state, path) ?? 0
if (typeof current !== 'number' || typeof value !== 'number') throw new TypeError(`$inc needs numbers at "${path}"`)
set[path] = current + value
break
}
case '$push':
set[path] = [...array(path), ...Json.clone(items(value))]
break
case '$addToSet':
set[path] = items(value).reduce<unknown[]>((acc, item) => (acc.some((x) => Json.equal(x, item)) ? acc : [...acc, Json.clone(item)]), array(path))
break
case '$pull':
set[path] = array(path).filter((item) => !DocumentQuery.matchesValue(item, value))
break
default:
throw new TypeError(`Unsupported update operator ${op}`)
}
}
}
return { op: 'patch', set, unset }
}
/** Applies a change to a state (`undefined` = no document). */
static apply(state: Doc | undefined, change: DocChange): Doc | undefined {
if (change.op === 'delete') return undefined
if (change.op === 'set') return Json.clone(change.doc)
const next = Json.clone(state ?? {})
for (const [path, value] of Object.entries(change.set)) DocumentPath.set(next, path, Json.clone(value))
for (const path of change.unset) DocumentPath.unset(next, path)
return next
}
}
src/utils/errors.ts
+19
View File
@@ -0,0 +1,19 @@
/** An optimistic-concurrency check failed (stale `expectedVersion` or a lost commit race). */
export class ConcurrencyError extends Error {
constructor(
readonly stream: string,
readonly expectedVersion: number | 'any',
readonly actualVersion: number,
) {
super(`Concurrency conflict on "${stream}": expected version ${expectedVersion}, actual ${actualVersion}`)
this.name = 'ConcurrencyError'
}
}
/** A key argument was passed but is `undefined`/`null`, empty, or not a string/number. */
export class InvalidKeyError extends TypeError {
constructor(message: string) {
super(message)
this.name = 'InvalidKeyError'
}
}
src/utils/json.ts
+24
View File
@@ -0,0 +1,24 @@
/** Helpers for JSON-compatible values (all stored values are JSON by contract). */
export class Json {
/** Deep copy; much cheaper than `structuredClone` for plain JSON. */
static clone<T>(value: T): T {
if (value === null || typeof value !== 'object') return value
if (Array.isArray(value)) return value.map((v) => Json.clone(v)) as T
const out: Record<string, unknown> = {}
for (const k in value) out[k] = Json.clone((value as Record<string, unknown>)[k])
return out as T
}
static equal(a: unknown, b: unknown): boolean {
if (a === b) return true
if (typeof a !== 'object' || typeof b !== 'object' || a === null || b === null) return false
if (Array.isArray(a) !== Array.isArray(b)) return false
const ka = Object.keys(a)
const kb = Object.keys(b)
return ka.length === kb.length && ka.every((k) => Json.equal((a as Record<string, unknown>)[k], (b as Record<string, unknown>)[k]))
}
static isObject(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
}
src/utils/keys.ts
+43
View File
@@ -0,0 +1,43 @@
import { InvalidKeyError } from './errors.js'
/**
* Key encoding. A node's key is its escaped path segments joined by `/`.
*
* Every character that sorts at or below `/` (and `~` itself) is escaped as `~XX` (hex). Escaped
* segments therefore only contain characters above `/`, so all keys of one child sort together:
* `p/a`, `p/a/...`, then siblings like `p/a~2D1` (`a-1`). That keeps child listing a simple skip-scan.
*
* Segments starting with `~` + a lowercase letter are reserved for internal use (e.g. `~e` for
* event commits); escaping only ever produces `~` + uppercase hex, so user keys can't collide.
*/
export class Keys {
static readonly separator = '/'
/** The character right after `/`: `prefix + segment + '0'` is the first key past a subtree. */
static readonly afterSeparator = '0'
/** Validates a user key and converts it to a segment. */
static segment(key: unknown): string {
if (key === undefined || key === null) throw new InvalidKeyError(`Key must not be ${key}`)
if (typeof key === 'number' && Number.isFinite(key)) return String(key)
if (typeof key !== 'string') throw new InvalidKeyError(`Key must be a string or finite number, got ${String(key)}`)
if (key === '') throw new InvalidKeyError('Key must not be empty')
return key
}
static escape(segment: string): string {
if (!/[\x00-\x2f~]/.test(segment)) return segment
return segment.replace(/[\x00-\x2f~]/g, (c) => '~' + c.charCodeAt(0).toString(16).toUpperCase().padStart(2, '0'))
}
static unescape(segment: string): string {
return segment.includes('~') ? segment.replace(/~([0-9A-F]{2})/g, (_, hex: string) => String.fromCharCode(parseInt(hex, 16))) : segment
}
static join(path: readonly string[]): string {
return path.map((s) => Keys.escape(s)).join(Keys.separator)
}
static isReserved(escapedSegment: string): boolean {
return /^~[a-z]/.test(escapedSegment)
}
}
src/types/schema.ts → src/utils/schema.ts
+4 -1
View File
@@ -15,13 +15,16 @@ export type ChildrenOf<S> = S extends { children: infer C } ? C : {}
export type KeyOf<S> = Extract<keyof ChildrenOf<S>, string | number>
export type ChildOf<S, K extends KeyOf<S>> = ChildrenOf<S>[K] extends infer C extends Spec ? C : never
/** Value of this node (`K = undefined`) or of child `K`. */
export type ValueAt<S, K> = [K] extends [KeyOf<S>] ? ValueOf<ChildOf<S, K>> : ValueOf<S>
/** Child keys as yielded at runtime (always strings). */
export type KeyString<S> = `${KeyOf<S>}`
/** `[key, value]` for each child key, as a union of precise pairs. */
export type EntryOf<S> = { [K in KeyOf<S>]: [K, ValueOf<ChildOf<S, K>>] }[KeyOf<S>]
/** `[key, value]` as yielded at runtime (string keys). Dynamic levels collapse to `[string, V]`. */
/** `[key, value]` as yielded at runtime (string keys). */
export type StringEntryOf<S> =
string extends KeyOf<S> ? [string, ValueOf<ChildOf<S, KeyOf<S> & string>>] : { [K in KeyOf<S>]: [`${K}`, ValueOf<ChildOf<S, K>>] }[KeyOf<S>]
src/utils/sorted-map.ts
+103
View File
@@ -0,0 +1,103 @@
/**
* Ordered string-keyed map: sorted chunks of at most `maxChunk` keys (split in half when full)
* plus a `Map` for values. Inserts and deletes cost O(log n + maxChunk).
*/
export class SortedMap<V> {
private chunks: string[][] = [[]]
private readonly values = new Map<string, V>()
/** Bumped on every insert/delete so iterators know to re-seek. */
private version = 0
constructor(private readonly maxChunk = 1024) {}
get size(): number {
return this.values.size
}
get(key: string): V | undefined {
return this.values.get(key)
}
has(key: string): boolean {
return this.values.has(key)
}
set(key: string, value: V): void {
if (!this.values.has(key)) {
const c = this.chunkFor(key)
const chunk = this.chunks[c]!
chunk.splice(SortedMap.lowerBound(chunk, key), 0, key)
if (chunk.length > this.maxChunk) this.chunks.splice(c + 1, 0, chunk.splice(this.maxChunk >>> 1))
this.version++
}
this.values.set(key, value)
}
delete(key: string): boolean {
if (!this.values.delete(key)) return false
const c = this.chunkFor(key)
const chunk = this.chunks[c]!
chunk.splice(SortedMap.lowerBound(chunk, key), 1)
if (chunk.length === 0 && this.chunks.length > 1) this.chunks.splice(c, 1)
this.version++
return true
}
/**
* Keys in `[gte, lt)`, descending when `reverse`. Safe across mutations between steps:
* after a change it re-seeks from the last key returned.
*/
*keys(gte?: string, lt?: string, reverse = false): IterableIterator<string> {
let last: string | undefined
let version = -1
let c = 0
let i = 0
for (;;) {
if (version !== this.version) {
version = this.version
const from = last ?? (reverse ? lt : gte)
if (from === undefined) {
c = reverse ? this.chunks.length - 1 : 0
i = reverse ? this.chunks[c]!.length - 1 : 0
} else {
c = this.chunkFor(from)
i = SortedMap.lowerBound(this.chunks[c]!, from)
// Forward: skip `last` itself. Reverse: step to the key before `from`.
if (reverse) i--
else if (last !== undefined && this.chunks[c]![i] === last) i++
}
}
// Move across chunk boundaries.
while (!reverse && c < this.chunks.length - 1 && i >= this.chunks[c]!.length) (c++, (i = 0))
while (reverse && c > 0 && i < 0) i = this.chunks[--c]!.length - 1
const key = this.chunks[c]?.[i]
if (key === undefined || (reverse ? gte !== undefined && key < gte : lt !== undefined && key >= lt)) return
last = key
yield key
i += reverse ? -1 : 1
}
}
/** Index of the last chunk whose first key is <= key (0 if none). */
private chunkFor(key: string): number {
let lo = 0
let hi = this.chunks.length - 1
while (lo < hi) {
const mid = (lo + hi + 1) >>> 1
if (this.chunks[mid]![0]! <= key) lo = mid
else hi = mid - 1
}
return lo
}
private static lowerBound(arr: string[], key: string): number {
let lo = 0
let hi = arr.length
while (lo < hi) {
const mid = (lo + hi) >>> 1
if (arr[mid]! < key) lo = mid + 1
else hi = mid
}
return lo
}
}
src/utils/ulid.ts
+36
View File
@@ -0,0 +1,36 @@
const ALPHABET = '0123456789ABCDEFGHJKMNPQRSTVWXYZ' // Crockford base32
/**
* ULID-like ids: 10 chars of millisecond time + 16 random chars, sortable by creation time.
* Ids from one generator within the same millisecond increment the random part, so they stay
* strictly increasing.
*/
export class Ulid {
private lastTime = -1
private readonly random = new Uint8Array(16) // one base32 digit (0–31) per byte
constructor(private readonly now: () => number = Date.now) {}
generate(): string {
let time = this.now()
if (time <= this.lastTime) {
time = this.lastTime
let i = 15
while (i >= 0 && this.random[i] === 31) this.random[i--] = 0
if (i < 0) throw new Error('ULID overflow within one millisecond')
this.random[i]!++
} else {
crypto.getRandomValues(this.random)
for (let i = 0; i < 16; i++) this.random[i]! &= 31
this.random[0]! &= 15 // headroom for increments
this.lastTime = time
}
let out = ''
for (let i = 0; i < 10; i++) {
out = ALPHABET[time % 32]! + out
time = Math.floor(time / 32)
}
for (const digit of this.random) out += ALPHABET[digit]
return out
}
}
test/core/encoding.test.ts
-73
View File
@@ -1,73 +0,0 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { decodeIndexValue, encodeIndexValue, type IndexValue } from '../../src/core/encoding.js'
const separators = fc.constantFrom('/', ':', '.', '|', '#', '!')
const indexValue: fc.Arbitrary<IndexValue> = fc.oneof(
fc.constant(null),
fc.boolean(),
fc.double({ noNaN: true }),
fc.integer(),
fc.string({ unit: 'binary' }),
fc.string({ unit: fc.constantFrom('/', ':', '0', '1', 'a', '%', '~', '\0', '!') }),
)
const rank = (v: IndexValue) => (v === null ? 0 : v === false ? 1 : v === true ? 2 : typeof v === 'number' ? 3 : 4)
function compare(a: IndexValue, b: IndexValue): number {
const ra = rank(a)
const rb = rank(b)
if (ra !== rb) return ra - rb
if (typeof a === 'number' && typeof b === 'number') return a === b ? 0 : a < b ? -1 : 1
if (typeof a === 'string' && typeof b === 'string') return a === b ? 0 : a < b ? -1 : 1
return 0
}
const sign = (n: number) => (n < 0 ? -1 : n > 0 ? 1 : 0)
const cmpStr = (a: string, b: string) => (a < b ? -1 : a > b ? 1 : 0)
describe('encodeIndexValue', () => {
it('round-trips (property)', () => {
fc.assert(
fc.property(indexValue, separators, (v, sep) => {
const decoded = decodeIndexValue(encodeIndexValue(v, sep), sep)
if (typeof v === 'number' && v === 0)
expect(decoded).toBe(0) // -0 normalised
else expect(decoded).toEqual(v)
}),
)
})
it('emits only characters above the separator (property)', () => {
fc.assert(
fc.property(indexValue, separators, (v, sep) => {
for (const c of encodeIndexValue(v, sep)) expect(c > sep).toBe(true)
}),
)
})
it('preserves order, including when followed by a separator and suffix (property)', () => {
fc.assert(
fc.property(indexValue, indexValue, fc.string(), fc.string(), separators, (a, b, sa, sb, sep) => {
const ea = encodeIndexValue(a, sep)
const eb = encodeIndexValue(b, sep)
const expected = sign(compare(a, b))
expect(cmpStr(ea, eb)).toBe(expected)
if (expected !== 0) expect(cmpStr(ea + sep + sa, eb + sep + sb)).toBe(expected)
}),
{ numRuns: 2000 },
)
})
it('orders types null < false < true < numbers < strings', () => {
const values: IndexValue[] = ['', 'a', -Infinity, -1, 0, 1e300, Infinity, true, false, null]
const sorted = [...values].sort((a, b) => cmpStr(encodeIndexValue(a), encodeIndexValue(b)))
expect(sorted).toEqual([null, false, true, -Infinity, -1, 0, 1e300, Infinity, '', 'a'])
})
it('rejects NaN and non-index values', () => {
expect(() => encodeIndexValue(NaN)).toThrow()
expect(() => encodeIndexValue({} as never)).toThrow()
})
})
test/core/path.test.ts
-73
View File
@@ -1,73 +0,0 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { escapeSegment, joinPath, pathKeys, splitPath, succ, toSegment, unescapeSegment, validateSeparator } from '../../src/core/path.js'
import { InvalidKeyError } from '../../src/errors.js'
const separators = fc.constantFrom('/', ':', '.', '|', '#', '\\', '!')
describe('escapeSegment', () => {
it('escapes %, the separator and a leading ~', () => {
expect(escapeSegment('a/b')).toBe('a%2Fb')
expect(escapeSegment('50%')).toBe('50%25')
expect(escapeSegment('~x')).toBe('%7Ex')
expect(escapeSegment('x~')).toBe('x~')
expect(escapeSegment('a:b', ':')).toBe('a%3Ab')
expect(escapeSegment('a/b', ':')).toBe('a/b')
expect(escapeSegment('plain')).toBe('plain')
})
it('round-trips (property)', () => {
fc.assert(
fc.property(fc.string({ unit: 'binary' }), separators, (s, sep) => {
expect(unescapeSegment(escapeSegment(s, sep))).toBe(s)
}),
)
})
it('never emits a raw separator or a leading ~ (property)', () => {
fc.assert(
fc.property(fc.string({ unit: 'binary' }), separators, (s, sep) => {
const e = escapeSegment(s, sep)
expect(e.includes(sep)).toBe(false)
expect(e.startsWith('~')).toBe(false)
}),
)
})
it('joins and splits paths (property)', () => {
fc.assert(
fc.property(fc.array(fc.string({ minLength: 1 }), { maxLength: 6 }), separators, (path, sep) => {
expect(splitPath(joinPath(path, sep), sep)).toEqual(path)
}),
)
})
})
describe('pathKeys', () => {
it('precomputes selfKey, prefix and subtree end', () => {
expect(pathKeys(['users', '123'], '/')).toMatchObject({ selfKey: 'users/123', prefix: 'users/123/', end: 'users/1230' })
expect(pathKeys([], '/')).toMatchObject({ selfKey: '', prefix: '', end: undefined })
expect(pathKeys(['a/b'], '/').selfKey).toBe('a%2Fb')
})
})
describe('misc', () => {
it('succ returns the next char', () => {
expect(succ('/')).toBe('0')
})
it('toSegment rejects undefined, null, empty and non-finite', () => {
expect(() => toSegment(undefined)).toThrow(InvalidKeyError)
expect(() => toSegment(null)).toThrow(TypeError)
expect(() => toSegment('')).toThrow(InvalidKeyError)
expect(() => toSegment(NaN)).toThrow(InvalidKeyError)
expect(toSegment(12)).toBe('12')
})
it('validates separators', () => {
expect(() => validateSeparator('//')).toThrow()
expect(() => validateSeparator('%')).toThrow()
expect(() => validateSeparator('~')).toThrow()
expect(() => validateSeparator(':')).not.toThrow()
})
})
test/document-storage/document-storage.suite.ts
+86
View File
@@ -0,0 +1,86 @@
import { describe, expect, it } from 'vitest'
import type { BaseDocumentStorage } from '../../src/document-storage/base-document-storage.js'
import { collect } from '../helpers.js'
type User = { name?: string; email?: string; age?: number; tags?: string[]; logins?: number; address?: { city?: string } }
/** Behaviour every document storage must have. `make` returns an empty storage. */
export function runDocumentStorageSuite(name: string, make: () => BaseDocumentStorage): void {
describe(`document storage: ${name}`, () => {
const users = () => make().child('users')
it('inserts with time-sortable ids, gets and lists', async () => {
const u = users()
const a = await u.insert({ name: 'Ada' })
const b = await u.insert({ name: 'Bob' })
expect(a < b).toBe(true)
expect(await u.get(a)).toEqual({ name: 'Ada' })
expect(await collect(u.keys())).toEqual([a, b])
})
it('set / get / delete in self and child forms', async () => {
const u = users()
await u.set('1', { name: 'Ada' })
expect(await u.child('1').get()).toEqual({ name: 'Ada' })
await u.child('1').set({ name: 'Ada L.' })
expect(await u.getMany(['1', '2'])).toEqual([{ name: 'Ada L.' }, undefined])
expect(await u.delete('1')).toBe(true)
expect(await u.child('1').delete()).toBe(false)
expect(await u.get('1')).toBeUndefined()
await u.set('1', { name: 'Again' })
expect(await u.get('1')).toEqual({ name: 'Again' })
})
it('updates with operators, returns the new state, and supports upsert', async () => {
const u = users()
await u.set('1', { name: 'Ada', tags: ['a'], logins: 1 })
expect(await u.update('1', { $inc: { logins: 2 }, $push: { tags: 'b' }, $set: { 'address.city': 'London' } })).toEqual({
name: 'Ada',
tags: ['a', 'b'],
logins: 3,
address: { city: 'London' },
})
expect(await u.child('1').update({ $unset: { address: 1 } })).toEqual({ name: 'Ada', tags: ['a', 'b'], logins: 3 })
expect(await u.update('missing', { $set: { a: 1 } })).toBeUndefined()
expect(await u.update('missing', { $inc: { n: 1 } }, { upsert: true })).toEqual({ n: 1 })
})
it('rejects bad keys and documents', async () => {
const u = users()
await expect(u.get(undefined as never)).rejects.toThrow(TypeError)
await expect(u.update(undefined as never, { $set: {} })).rejects.toThrow(TypeError)
await expect(make().get()).rejects.toThrow(/root/)
await expect(u.set('x', [] as never)).rejects.toThrow(/plain object/)
})
it('nests documents under documents', async () => {
const u = users()
await u.set('1', { name: 'Ada' })
await u.child('1').child('posts').set('p1', { title: 'Hi' })
expect(await u.child('1').child('posts').get('p1')).toEqual({ title: 'Hi' })
expect((await collect(u.find())).map((r) => r.key)).toEqual(['1'])
})
it('finds with equality, comparisons, $in, dot paths, sort, skip and limit', async () => {
const u = users()
const rows: [string, User][] = [
['a', { name: 'Ada', age: 36, email: 'ada@x', tags: ['x', 'y'], address: { city: 'London' } }],
['b', { name: 'Bob', age: 20, email: 'bob@x', tags: ['y'] }],
['c', { name: 'Cy', age: 51, address: { city: 'London' } }],
['d', { name: 'Di', age: 20 }],
]
for (const [k, v] of rows) await u.set(k, v)
const keys = async (filter: Record<string, unknown>, options = {}) => (await collect(u.find(filter, options))).map((r) => r.key)
expect(await keys({ age: 20 })).toEqual(['b', 'd'])
expect(await keys({ age: { $gte: 20, $lt: 40 } })).toEqual(['a', 'b', 'd'])
expect(await keys({ age: { $in: [36, 51] } })).toEqual(['a', 'c'])
expect(await keys({ 'address.city': 'London', age: { $gt: 40 } })).toEqual(['c'])
expect(await keys({ tags: 'y' })).toEqual(['a', 'b'])
expect(await keys({ email: null })).toEqual(['c', 'd'])
expect(await keys({}, { sort: { age: -1 } })).toEqual(['c', 'a', 'b', 'd'])
expect(await keys({}, { sort: { age: 1 }, skip: 1, limit: 2 })).toEqual(['d', 'a'])
expect(await u.findOne({ email: 'bob@x' })).toEqual({ key: 'b', data: rows[1]![1] })
expect(await u.findOne({ email: 'nobody' })).toBeUndefined()
})
})
}
test/document-storage/document-storage.test.ts
+91
View File
@@ -0,0 +1,91 @@
import { describe, expect, it } from 'vitest'
import { DocumentStorageInMemory } from '../../src/document-storage/document-storage-in-memory.js'
import { DocumentStorageKV } from '../../src/document-storage/document-storage-kv.js'
import { EventSourcedDocumentStorage } from '../../src/document-storage/event-sourced-document-storage.js'
import { EventStorageKV } from '../../src/event-storage/event-storage-kv.js'
import { KVStorageInMemory } from '../../src/kv-storage/kv-storage-in-memory.js'
import { collect, KVStorageCounting } from '../helpers.js'
import { runDocumentStorageSuite } from './document-storage.suite.js'
const indexes = { users: ['email', 'age'] }
runDocumentStorageSuite('DocumentStorageInMemory', () => new DocumentStorageInMemory())
runDocumentStorageSuite('DocumentStorageInMemory (indexed)', () => new DocumentStorageInMemory({ indexes }))
runDocumentStorageSuite('DocumentStorageKV', () => new DocumentStorageKV(new KVStorageInMemory()))
runDocumentStorageSuite('EventSourced (in-memory projections)', () => {
const kv = new KVStorageInMemory()
return new EventSourcedDocumentStorage(new EventStorageKV(kv), new DocumentStorageInMemory({ indexes }))
})
runDocumentStorageSuite('EventSourced (KV projections)', () => {
const kv = new KVStorageInMemory()
return new EventSourcedDocumentStorage(new EventStorageKV(kv), new DocumentStorageKV(kv.child('projections')))
})
describe('DocumentStorageInMemory indexes', () => {
it('keeps the equality index in sync with updates and deletes', async () => {
const users = new DocumentStorageInMemory({ indexes }).child('users')
await users.set('1', { email: 'a', age: 1 })
await users.set('2', { email: 'b', age: 1 })
await users.update('1', { $set: { email: 'c' } })
await users.delete('2')
expect(await collect(users.find({ email: 'a' }))).toEqual([])
expect(await collect(users.find({ email: 'c' }))).toEqual([{ key: '1', data: { email: 'c', age: 1 } }])
expect(await collect(users.find({ age: { $eq: 1 } }))).toHaveLength(1)
})
})
describe('EventSourcedDocumentStorage', () => {
function setup() {
const kv = new KVStorageCounting()
const events = new EventStorageKV(kv)
const projections = new DocumentStorageKV(kv.child('projections'))
return { kv, events, projections, docs: new EventSourcedDocumentStorage(events, projections) }
}
it('records resolved changes as history and replays any version', async () => {
const users = setup().docs.child('users')
await users.set('1', { n: 5 })
await users.update('1', { $inc: { n: 1 } })
await users.delete('1')
await users.set('1', { n: 10 })
expect((await collect(users.history('1'))).map((e) => e.data)).toEqual([
{ op: 'set', doc: { n: 5 } },
{ op: 'patch', set: { n: 6 }, unset: [] },
{ op: 'delete' },
{ op: 'set', doc: { n: 10 } },
])
expect(await users.getAt('1', 2)).toEqual({ n: 6 })
expect(await users.child('1').getAt(3)).toBeUndefined()
expect(await users.get('1')).toEqual({ n: 10 })
})
it('heals a projection left behind by a crash between append and projection write', async () => {
const { docs, events } = setup()
const users = docs.child('users')
await users.set('1', { n: 1 })
await events.child('users').append('1', [{ op: 'patch', set: { n: 2 }, unset: [] }]) // projection never written
expect(await users.get('1')).toEqual({ n: 1 })
expect(await users.update('1', { $inc: { n: 1 } })).toEqual({ n: 3 })
await events.child('users').append('2', [{ op: 'set', doc: { n: 7 } }]) // first write crashed
expect(await users.catchUp('2')).toEqual({ n: 7 })
expect(await users.get('2')).toEqual({ n: 7 })
})
it('two instances incrementing the same document converge', async () => {
const kv = new KVStorageInMemory()
const make = () => new EventSourcedDocumentStorage(new EventStorageKV(kv), new DocumentStorageKV(kv.child('projections'))).child('users')
const a = make()
const b = make()
await a.set('1', { n: 0 })
await Promise.all([...Array(5)].flatMap(() => [a.update('1', { $inc: { n: 1 } }), b.update('1', { $inc: { n: 1 } })]))
expect(await a.get('1')).toEqual({ n: 10 })
})
it('a write is a projection read, an append and a projection write', async () => {
const { docs, kv } = setup()
await docs.child('users').set('1', { n: 1 })
kv.reset()
await docs.child('users').update('1', { $inc: { n: 1 } })
expect(kv.calls).toEqual({ readMany: 2, writeIfAbsent: 1, writeMany: 1 })
})
})
test/documents/event-sourced.test.ts
-379
View File
@@ -1,379 +0,0 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import fc from 'fast-check'
import { KVStorage } from '../../src/kv/node.js'
import { InMemoryKVDriver } from '../../src/kv/memory.js'
import { KVEventStorage } from '../../src/events/kv.js'
import { EventSourcedDocumentStorage, type EventSourcedDocumentStorageOptions } from '../../src/documents/event-sourced.js'
import type { DocIndexDeclarations } from '../../src/documents/indexes.js'
import type { Filter } from '../../src/documents/filter.js'
import { CapabilityError, UniqueConstraintError } from '../../src/errors.js'
import { collect, CountingDriver, type Capability } from '../helpers.js'
type User = { name?: string; email?: string; age?: number; tags?: string[]; logins?: number; address?: { city?: string } }
const userIndexes: DocIndexDeclarations = {
users: [{ fields: ['email'], unique: true }, { fields: ['age'] }, { name: 'cityAge', fields: ['address.city', 'age'] }, { fields: ['tags'] }],
'users/*/posts': [{ fields: ['createdAt'] }],
}
function setup(opts: { hide?: Capability[]; inner?: InMemoryKVDriver } & EventSourcedDocumentStorageOptions = {}) {
const { hide = [], inner = new InMemoryKVDriver(), ...docOpts } = opts
const driver = new CountingDriver(inner, hide)
const kv = new KVStorage(driver)
const events = new KVEventStorage(kv)
const docs = new EventSourcedDocumentStorage(events, { indexes: userIndexes, warnOnFullScan: false, ...docOpts })
return { driver, kv, events, docs, users: docs.child('users') }
}
afterEach(() => vi.restoreAllMocks())
describe.each([
['atomic batch', [] as Capability[]],
['non-atomic', ['batch'] as Capability[]],
])('EventSourcedDocumentStorage (%s)', (_, hide) => {
describe('CRUD', () => {
it('inserts with time-sortable ids and reads back', async () => {
const { users } = setup({ hide })
const a = await users.insert({ name: 'Ada', age: 36 })
const b = await users.insert({ name: 'Bob', age: 20 })
expect(a).toMatch(/^[0-9A-HJKMNP-TV-Z]{26}$/)
expect(a < b).toBe(true)
expect(await users.get(a)).toEqual({ name: 'Ada', age: 36 })
expect(await collect(users.keys())).toEqual([a, b])
})
it('set / get / delete in self and child forms', async () => {
const { users } = setup({ hide })
await users.set('123', { name: 'Ada', age: 36 })
expect(await users.child('123').get()).toEqual({ name: 'Ada', age: 36 })
await users.child('123').set({ name: 'Ada L.' })
expect(await users.get('123')).toEqual({ name: 'Ada L.' })
expect(await users.delete('123')).toBe(true)
expect(await users.delete('123')).toBe(false)
expect(await users.get('123')).toBeUndefined()
expect(await collect(users.keys())).toEqual([])
await users.set('123', { name: 'Again' })
expect(await users.get('123')).toEqual({ name: 'Again' })
})
it('applies arity rules', async () => {
const { users, docs } = setup({ hide })
const id: string | undefined = undefined
await expect(users.get(id as never)).rejects.toThrow(TypeError)
await expect(users.update(id as never, { $set: { a: 1 } })).rejects.toThrow(TypeError)
await expect(users.delete(id as never)).rejects.toThrow(TypeError)
await expect(docs.get()).rejects.toThrow(/root/)
await expect(users.set('x', [] as never)).rejects.toThrow(/plain object/)
})
it('nests documents under documents', async () => {
const { users } = setup({ hide })
await users.set('u1', { name: 'Ada' })
const posts = users.child('u1').child('posts')
await posts.set('p1', { title: 'Hello', createdAt: '2024-01' })
expect(await posts.get('p1')).toEqual({ title: 'Hello', createdAt: '2024-01' })
expect((await collect(users.find())).map((r) => r.key)).toEqual(['u1'])
expect((await collect(posts.find({ createdAt: { $gte: '2024' } }))).map((r) => r.key)).toEqual(['p1'])
})
})
describe('update', () => {
it('supports $set $unset $inc $push $pull $addToSet with dot paths', async () => {
const { users } = setup({ hide })
await users.set('1', { name: 'Ada', tags: ['a'], address: { city: 'X' }, nickname: 'A' } as User)
const r = await users.update('1', {
$set: { name: 'Ada L.', 'address.zip': '123' },
$inc: { logins: 1 },
$push: { tags: { $each: ['b', 'c'] } },
$unset: { nickname: true },
})
expect(r).toEqual({ name: 'Ada L.', tags: ['a', 'b', 'c'], address: { city: 'X', zip: '123' }, logins: 1 })
await users.update('1', { $pull: { tags: { $in: ['a', 'c'] } }, $addToSet: { 'address.list': { $each: [1, 1, 2] } } })
await users.child('1').update({ $inc: { logins: 5 }, $addToSet: { tags: 'b' } })
expect(await users.get('1')).toEqual({ name: 'Ada L.', tags: ['b'], address: { city: 'X', zip: '123', list: [1, 2] }, logins: 6 })
})
it('returns undefined for missing docs unless upsert', async () => {
const { users } = setup({ hide })
expect(await users.update('nope', { $set: { a: 1 } })).toBeUndefined()
expect(await users.get('nope')).toBeUndefined()
expect(await users.update('nope', { $inc: { n: 2 } }, { upsert: true })).toEqual({ n: 2 })
})
it('rejects conflicting paths and bad operators', async () => {
const { users } = setup({ hide })
await users.set('1', { age: 1 })
await expect(users.update('1', { $set: { a: 1 }, $unset: { 'a.b': 1 } })).rejects.toThrow(/conflict/)
await expect(users.update('1', { $rename: {} } as never)).rejects.toThrow(/Unsupported/)
await expect(users.update('1', { $inc: { name: 1 } }).then(() => users.update('1', { $push: { age: 1 } }))).rejects.toThrow(/non-array/)
})
it('stores resolved deltas, not operators, and skips no-op updates', async () => {
const { users } = setup({ hide })
await users.set('1', { logins: 5 })
await users.update('1', { $inc: { logins: 1 } })
await users.update('1', { $set: { logins: 6 } }) // no-op
expect((await collect(users.history('1'))).map((e) => e.data)).toEqual([
{ op: 'set', doc: { logins: 5 } },
{ op: 'patch', set: { logins: 6 } },
])
})
})
describe('history and getAt', () => {
it('replays to any version', async () => {
const { users } = setup({ hide })
await users.set('1', { n: 1 })
await users.update('1', { $inc: { n: 1 } })
await users.delete('1')
await users.set('1', { n: 10 })
expect(await users.getAt('1', 0)).toBeUndefined()
expect(await users.getAt('1', 1)).toEqual({ n: 1 })
expect(await users.getAt('1', 2)).toEqual({ n: 2 })
expect(await users.getAt('1', 3)).toBeUndefined()
expect(await users.child('1').getAt(4)).toEqual({ n: 10 })
expect(await users.getAt('1', 99)).toEqual({ n: 10 })
expect((await collect(users.child('1').history())).map((e) => e.version)).toEqual([1, 2, 3, 4])
})
})
describe('find', () => {
async function seed() {
const ctx = setup({ hide })
const rows: [string, User][] = [
['a', { name: 'Ada', age: 36, email: 'ada@x', tags: ['x', 'y'], address: { city: 'London' } }],
['b', { name: 'Bob', age: 20, email: 'bob@x', tags: ['y'], address: { city: 'Paris' } }],
['c', { name: 'Cy', age: 51, tags: [], address: { city: 'London' } }],
['d', { name: 'Di', age: 20, email: 'di@x' }],
['e', { name: 'Ed', email: 'ed@x', address: { city: 'London' } }],
]
for (const [k, v] of rows) await ctx.users.set(k, v)
await ctx.users.set('gone', { name: 'Gone', age: 20 })
await ctx.users.delete('gone')
return ctx
}
const keysOf = async (it: AsyncIterable<{ key: string }>) => (await collect(it)).map((r) => r.key)
it('evaluates comparison, logical and element operators', async () => {
const { users } = await seed()
// Unsorted results come in index or key order depending on the plan; compare as sets.
const q = async (f: Filter) => (await keysOf(users.find(f))).sort()
expect(await q({})).toEqual(['a', 'b', 'c', 'd', 'e'])
expect(await q({ age: 20 })).toEqual(['b', 'd'])
expect(await q({ age: { $gte: 20, $lt: 40 } })).toEqual(['a', 'b', 'd'])
expect(await q({ age: { $gt: 20 } })).toEqual(['a', 'c'])
expect(await q({ age: { $ne: 20 } })).toEqual(['a', 'c', 'e'])
expect(await q({ age: { $in: [36, 51] } })).toEqual(['a', 'c'])
expect(await q({ age: { $nin: [36, 51] } })).toEqual(['b', 'd', 'e'])
expect(await q({ age: { $exists: false } })).toEqual(['e'])
expect(await q({ age: null })).toEqual(['e'])
expect(await q({ email: { $exists: true }, age: { $lte: 20 } })).toEqual(['b', 'd'])
expect(await q({ 'address.city': 'London' })).toEqual(['a', 'c', 'e'])
expect(await q({ 'address.city': 'London', age: { $gt: 40 } })).toEqual(['c'])
expect(await q({ tags: 'y' })).toEqual(['a', 'b'])
expect(await q({ tags: ['y'] })).toEqual(['b'])
expect(await q({ $or: [{ age: 51 }, { name: 'Bob' }] })).toEqual(['b', 'c'])
expect(await q({ $and: [{ age: { $gte: 20 } }, { age: { $lte: 20 } }] })).toEqual(['b', 'd'])
expect(await q({ $not: { age: 20 } })).toEqual(['a', 'c', 'e'])
expect(await q({ age: { $not: { $gte: 30 } } })).toEqual(['b', 'd', 'e'])
expect(await q({ name: { $gt: 5 } })).toEqual([]) // type-bracketed
})
it('sorts (index-backed and in-memory), skips, limits and projects', async () => {
const { users } = await seed()
expect(await keysOf(users.find({}, { sort: { age: 1 } }))).toEqual(['e', 'b', 'd', 'a', 'c'])
expect(await keysOf(users.find({}, { sort: { age: -1 } }))).toEqual(['c', 'a', 'd', 'b', 'e'])
expect(await keysOf(users.find({}, { sort: { name: -1 }, skip: 1, limit: 2 }))).toEqual(['d', 'c'])
expect(await keysOf(users.find({ age: { $gte: 20 } }, { sort: { age: -1 }, limit: 2 }))).toEqual(['c', 'a'])
expect(await keysOf(users.find({ 'address.city': 'London' }, { sort: { age: 1 } }))).toEqual(['e', 'a', 'c'])
expect(await collect(users.find({ age: 36 }, { projection: { name: 1, 'address.city': 1 } }))).toEqual([
{ key: 'a', data: { name: 'Ada', address: { city: 'London' } } },
])
expect(await collect(users.find({ age: 36 }, { projection: { tags: 0, address: 0, email: 0 } }))).toEqual([{ key: 'a', data: { name: 'Ada', age: 36 } }])
expect(await users.findOne({ email: 'bob@x' })).toEqual({ key: 'b', data: expect.objectContaining({ name: 'Bob' }) })
expect(await users.findOne({ email: 'nobody' })).toBeUndefined()
})
it('uses an index: findOne costs one entry scan and one projection read', async () => {
const { users, driver } = await seed()
driver.reset()
await users.findOne({ email: 'di@x' })
expect(driver.calls).toEqual({ scan: 1, getMany: 1 })
})
it('warns once per filter shape on full scans', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
const { users } = setup({ hide, warnOnFullScan: true })
await collect(users.find({ name: 'x' }))
await collect(users.find({ name: 'y' }))
await collect(users.find({ age: 1 }))
expect(warn).toHaveBeenCalledTimes(1)
expect(warn.mock.calls[0]![0]).toMatch(/full scan for filter \{"name":"\?"\}/)
})
})
describe('unique indexes', () => {
it('rejects duplicates and frees values on change and delete', async () => {
const { users } = setup({ hide })
await users.set('1', { email: 'a@x' })
await expect(users.set('2', { email: 'a@x' })).rejects.toBeInstanceOf(UniqueConstraintError)
await expect(users.insert({ email: 'a@x' })).rejects.toMatchObject({ index: 'email', existingKey: '1' })
expect(await collect(users.keys())).toEqual(['1'])
await users.update('1', { $set: { email: 'b@x' } })
await users.set('2', { email: 'a@x' })
await users.delete('2')
await users.set('3', { email: 'a@x' })
await users.set('4', {}) // missing values are not claimed
await users.set('5', {})
await users.set('3', { email: 'a@x', name: 'same value, same doc' })
})
it('exactly one of concurrent writers claiming the same value wins', async () => {
const { users } = setup({ hide })
const results = await Promise.allSettled(['1', '2', '3'].map((k) => users.set(k, { email: 'same' })))
expect(results.filter((r) => r.status === 'fulfilled')).toHaveLength(1)
expect((await collect(users.find({ email: 'same' }))).length).toBe(1)
})
it('takes over a stale claim only after the grace period', async () => {
let t = 0
const { users, driver } = setup({ hide, now: () => t, claimGraceMs: 1000 })
// Leave an orphaned claim: doc 1's projection and events vanish but its claim remains.
await users.set('1', { email: 'a@x' })
await driver.inner.deleteMany(['~p/users/1'])
await driver.inner.deleteMany((await collect(driver.inner.scan({ gte: 'users/1/', lt: 'users/10' }))).map(([k]) => k))
await expect(users.set('2', { email: 'a@x' })).rejects.toBeInstanceOf(UniqueConstraintError)
t = 5000
await users.set('2', { email: 'a@x' })
expect(await users.get('2')).toEqual({ email: 'a@x' })
})
it('requires setIfAbsent', () => {
const kv = new KVStorage(new CountingDriver(new InMemoryKVDriver(), ['setIfAbsent']))
expect(() => new EventSourcedDocumentStorage(new KVEventStorage(kv), { indexes: userIndexes })).toThrow(CapabilityError)
})
})
describe('projection lag (crash between append and projection write)', () => {
async function lagged() {
const ctx = setup({ hide })
await ctx.users.set('1', { n: 1, email: 'a@x' })
// Simulate a crash: the event is appended but the projection is never written.
await ctx.events.child('users').append('1', [{ op: 'patch', set: { n: 2, email: 'b@x' } }])
return ctx
}
it('reads see the stale projection; consistent reads catch up', async () => {
const { users } = await lagged()
expect(await users.get('1')).toEqual({ n: 1, email: 'a@x' })
expect(await users.get('1', { consistent: true })).toEqual({ n: 2, email: 'b@x' })
expect(await users.findOne({ email: 'b@x' })).toMatchObject({ key: '1' })
expect(await users.findOne({ email: 'a@x' })).toBeUndefined()
})
it('the next write self-heals', async () => {
const { users } = await lagged()
expect(await users.update('1', { $inc: { n: 1 } })).toEqual({ n: 3, email: 'b@x' })
expect((await collect(users.history('1'))).map((e) => e.version)).toEqual([1, 2, 3])
await users.set('2', { email: 'a@x' }) // old claim was released by the catch-up
})
it('catchUp() repairs a subtree', async () => {
const { users, docs, events } = await lagged()
await events
.child('users')
.child('9')
.child('posts')
.append('p', [{ op: 'set', doc: { createdAt: 'x' } }])
expect(await docs.catchUp()).toBe(2)
expect(await users.get('1')).toEqual({ n: 2, email: 'b@x' })
expect(await users.child('9').child('posts').get('p')).toEqual({ createdAt: 'x' })
expect(await docs.catchUp()).toBe(0)
})
})
describe('concurrency', () => {
it('two storage instances incrementing the same doc converge', async () => {
const inner = new InMemoryKVDriver()
const a = setup({ hide, inner }).users
const b = setup({ hide, inner }).users
await a.set('1', { n: 0 })
await Promise.all([...Array(5)].flatMap(() => [a.update('1', { $inc: { n: 1 } }), b.update('1', { $inc: { n: 1 } })]))
expect(await a.get('1')).toEqual({ n: 10 })
expect(await b.get('1', { consistent: true })).toEqual({ n: 10 })
})
})
})
describe('costs', () => {
it('get is one driver call; update is projection read + append + projection write, even cold', async () => {
const inner = new InMemoryKVDriver()
await setup({ inner }).users.set('1', { name: 'a', age: 1 })
const { users, driver } = setup({ inner }) // fresh instance: cold head cache
driver.reset()
await users.get('1')
expect(driver.calls).toEqual({ getMany: 1 })
driver.reset()
await users.update('1', { $set: { age: 2 } })
expect(driver.calls).toEqual({ getMany: 1, setIfAbsent: 1, batch: 1 })
})
})
describe('property: indexed find equals unindexed find', () => {
const doc = fc.record(
{
age: fc.option(fc.integer({ min: 0, max: 5 }), { nil: undefined }),
email: fc.constantFrom('a', 'b', 'c', 'd', 'e', 'f'),
tags: fc.array(fc.constantFrom('x', 'y', 'z'), { maxLength: 3 }),
address: fc.option(fc.record({ city: fc.constantFrom('L', 'P') }), { nil: undefined }),
},
{ requiredKeys: [] },
)
const bound = fc.integer({ min: -1, max: 6 })
const filter: fc.Arbitrary<Filter> = fc.oneof(
fc.record({ age: bound }),
fc.record({ age: fc.record({ $gte: bound, $lt: bound }, { requiredKeys: [] }) }),
fc.record({ age: fc.record({ $gt: bound, $lte: bound }, { requiredKeys: [] }) }),
fc.record({ tags: fc.constantFrom('x', 'y', 'z') }),
fc.record({ tags: fc.record({ $gte: fc.constantFrom('x', 'y'), $lt: fc.constantFrom('y', 'zz') }) }),
fc.record({ 'address.city': fc.constantFrom('L', 'P'), age: fc.record({ $gt: bound }) }),
fc.record({ 'address.city': fc.constantFrom('L', 'P') }),
fc.constant({ age: null }),
fc.constant({}),
)
const sort = fc.option(fc.record({ age: fc.constantFrom(1 as const, -1 as const) }), { nil: undefined })
it('returns the same results in the same order', async () => {
await fc.assert(
fc.asyncProperty(
fc.array(fc.tuple(fc.constantFrom('1', '2', '3', '4', '5'), doc), { maxLength: 12 }),
filter,
sort,
fc.option(fc.integer({ min: 1, max: 4 }), { nil: undefined }),
async (writes, f, s, limit) => {
const plain = new EventSourcedDocumentStorage(new KVEventStorage(new KVStorage(new InMemoryKVDriver())), { warnOnFullScan: false }).child('users')
const indexed = new EventSourcedDocumentStorage(new KVEventStorage(new KVStorage(new InMemoryKVDriver())), {
warnOnFullScan: false,
indexes: { users: userIndexes.users!.map((i) => ({ ...i, unique: false })) },
}).child('users')
for (const [k, v] of writes) {
await plain.set(k, v)
await indexed.set(k, v)
}
const opts = { ...(s ? { sort: s } : {}), ...(limit ? { limit } : {}) }
const a = await collect(plain.find(f, opts))
const b = await collect(indexed.find(f, opts))
if (s) {
expect(b).toEqual(a)
return
}
// Unsorted order depends on the plan: compare as sets (a limited result is any subset of that size).
const all = new Set((await collect(plain.find(f))).map((r) => r.key))
expect(b).toHaveLength(a.length)
for (const r of b) expect(all.has(r.key)).toBe(true)
},
),
{ numRuns: 300 },
)
})
})
test/documents/units.test.ts
-62
View File
@@ -1,62 +0,0 @@
import { describe, expect, it } from 'vitest'
import { createIdGenerator } from '../../src/documents/ulid.js'
import { matches } from '../../src/documents/filter.js'
import { applyPatch, resolveUpdate } from '../../src/documents/update.js'
describe('ulid', () => {
it('is strictly increasing within one millisecond and sorts by time', () => {
let t = 1_700_000_000_000
const gen = createIdGenerator(() => t)
const ids = Array.from({ length: 1000 }, () => gen())
expect([...ids].sort()).toEqual(ids)
expect(new Set(ids).size).toBe(1000)
t += 1
const later = gen()
expect(later > ids.at(-1)!).toBe(true)
expect(later.slice(0, 10) > ids[0]!.slice(0, 10)).toBe(true)
})
})
describe('filter semantics', () => {
it('requires one array element to satisfy all positive operators of a condition', () => {
expect(matches({ s: [1, 20] }, { s: { $gt: 5, $lt: 10 } })).toBe(false) // MongoDB would match
expect(matches({ s: [1, 7] }, { s: { $gt: 5, $lt: 10 } })).toBe(true)
expect(matches({ s: [1, 20] }, { $and: [{ s: { $gt: 5 } }, { s: { $lt: 10 } }] })).toBe(true)
})
it('traverses arrays of sub-documents on dot paths', () => {
expect(matches({ a: [{ b: 1 }, { b: 2 }] }, { 'a.b': 2 })).toBe(true)
expect(matches({ a: [{ b: 1 }] }, { 'a.0.b': 1 })).toBe(true)
expect(matches({ a: [{ b: 1 }] }, { 'a.b': { $exists: true } })).toBe(true)
expect(matches({ a: [] }, { 'a.b': { $exists: true } })).toBe(false)
})
it('treats null as matching missing', () => {
expect(matches({}, { x: null })).toBe(true)
expect(matches({ x: null }, { x: { $ne: null } })).toBe(false)
expect(matches({ x: 0 }, { x: { $ne: null } })).toBe(true)
})
it('rejects unknown operators', () => {
expect(() => matches({}, { x: { $regex: 'a' } })).toThrow(/Unknown operator/)
expect(() => matches({}, { $nor: [] } as never)).toThrow(/Unknown top-level/)
})
})
describe('update resolution', () => {
it('$pull by sub-document filter and by condition', () => {
const state = {
items: [
{ k: 1, v: 'a' },
{ k: 2, v: 'b' },
],
n: [1, 5, 9],
}
const patch = resolveUpdate(state, { $pull: { items: { k: 2 }, n: { $gte: 5 } } })
expect(applyPatch(state, patch)).toEqual({ items: [{ k: 1, v: 'a' }], n: [1] })
})
it('$unset of an array position nulls it', () => {
expect(applyPatch({ a: [1, 2] }, resolveUpdate({ a: [1, 2] }, { $unset: { 'a.0': true } }))).toEqual({ a: [null, 2] })
})
})
test/event-storage/event-storage-kv.test.ts
+99
View File
@@ -0,0 +1,99 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { EventStorageKV } from '../../src/event-storage/event-storage-kv.js'
import { KVStorageInMemory } from '../../src/kv-storage/kv-storage-in-memory.js'
import { ConcurrencyError } from '../../src/utils/errors.js'
import { collect, KVStorageCounting } from '../helpers.js'
function setup(kv = new KVStorageCounting()) {
let t = 1000
return { kv, events: new EventStorageKV(kv, { now: () => t++ }) }
}
describe('EventStorageKV', () => {
it('appends and reads in self and child forms', async () => {
const orders = setup().events.child('orders')
expect(await orders.append('o1', [{ type: 'created' }], { expectedVersion: 0 })).toEqual({ version: 1 })
expect(await orders.child('o1').append([{ type: 'paid' }, { type: 'shipped' }])).toEqual({ version: 3 })
expect(await collect(orders.read('o1'))).toEqual([
{ version: 1, at: 1000, data: { type: 'created' } },
{ version: 2, at: 1001, data: { type: 'paid' } },
{ version: 3, at: 1001, data: { type: 'shipped' } },
])
expect(await collect(orders.child('o1').read({ from: 3 }))).toHaveLength(1)
expect(await orders.version('o1')).toBe(3)
expect(await orders.version('none')).toBe(0)
})
it('stores one key per commit', async () => {
const { kv, events } = setup()
await events.child('s').append(['a'])
await events.child('s').append(['b', 'c'])
expect((await collect(kv.scan())).map(([k]) => k)).toEqual(['s/~e/000000000001', 's/~e/000000000002'])
})
it('enforces expectedVersion and the arity rules', async () => {
const { events } = setup()
const s = events.child('s')
await s.append(['a'])
await expect(s.append(['b'], { expectedVersion: 0 })).rejects.toBeInstanceOf(ConcurrencyError)
await expect(events.append(['x'])).rejects.toThrow(/root/)
await expect(events.child('o').append(undefined as never, ['x'] as never)).rejects.toThrow(TypeError)
await expect(events.child('o').version(undefined as never)).rejects.toThrow(TypeError)
})
it('lists child streams', async () => {
const orders = setup().events.child('orders')
await orders.append('o1', ['a'])
await orders.append('o2', ['b'])
await orders.child('o1').append('notes', ['n'])
expect(await collect(orders.keys())).toEqual(['o1', 'o2'])
expect(await collect(orders.child('o1').keys())).toEqual(['notes'])
})
it('a warm append is one writeIfAbsent; a cold one adds one reverse scan', async () => {
const { kv, events } = setup()
await events.child('s').append(['a'])
kv.reset()
await events.child('s').append(['b'])
expect(kv.calls).toEqual({ writeIfAbsent: 1 })
const cold = setup(kv).events
kv.reset()
await cold.child('s').append(['c'])
expect(kv.calls).toEqual({ scan: 1, writeIfAbsent: 1 })
})
it('a writer with a stale cache collides; with "any" it retries', async () => {
const kv = new KVStorageInMemory()
const a = setup(new KVStorageCounting(kv)).events.child('s')
const b = setup(new KVStorageCounting(kv)).events.child('s')
await a.append(['a1'])
await b.append(['b1'])
await expect(a.append(['a2'], { expectedVersion: 1 })).rejects.toMatchObject({ actualVersion: 2 })
expect(await a.append(['a3'])).toEqual({ version: 3 })
const results = await Promise.allSettled([a.append(['x'], { expectedVersion: 3 }), b.append(['y'], { expectedVersion: 3 })])
expect(results.filter((r) => r.status === 'fulfilled')).toHaveLength(1)
})
it('reads any window correctly (property)', async () => {
await fc.assert(
fc.asyncProperty(
fc.array(fc.integer({ min: 1, max: 4 }), { maxLength: 20 }),
fc.record({
from: fc.option(fc.integer({ min: 0, max: 60 }), { nil: undefined }),
to: fc.option(fc.integer({ min: 0, max: 60 }), { nil: undefined }),
limit: fc.option(fc.nat(10), { nil: undefined }),
}),
async (sizes, options) => {
const s = setup().events.child('s')
let n = 0
for (const size of sizes) await s.append(Array.from({ length: size }, () => ++n))
let expected = Array.from({ length: n }, (_, i) => i + 1).filter((v) => v >= (options.from ?? 1) && v <= (options.to ?? Infinity))
if (options.limit !== undefined) expected = expected.slice(0, options.limit)
expect((await collect(s.read(options))).map((e) => e.data)).toEqual(expected)
},
),
{ numRuns: 200 },
)
})
})
test/events/kv.test.ts
-200
View File
@@ -1,200 +0,0 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { KVStorage } from '../../src/kv/node.js'
import { InMemoryKVDriver } from '../../src/kv/memory.js'
import { KVEventStorage } from '../../src/events/kv.js'
import { ConcurrencyError } from '../../src/errors.js'
import { collect, CountingDriver, type Capability } from '../helpers.js'
function setup(hide: Capability[] = [], inner = new InMemoryKVDriver()) {
const driver = new CountingDriver(inner, hide)
let t = 1000
const events = new KVEventStorage(new KVStorage(driver), { now: () => t++ })
return { driver, events }
}
describe('KVEventStorage', () => {
it('appends and reads with versions and metadata', async () => {
const { events } = setup()
const orders = events.child('orders')
expect(await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 })).toEqual({ version: 1, commit: 1 })
expect(await orders.child('o1').append([{ type: 'paid' }, { type: 'shipped' }])).toEqual({ version: 3, commit: 2 })
expect(await collect(orders.read('o1'))).toEqual([
{ version: 1, commit: 1, at: 1000, data: { type: 'created', total: 40 } },
{ version: 2, commit: 2, at: 1001, data: { type: 'paid' } },
{ version: 3, commit: 2, at: 1001, data: { type: 'shipped' } },
])
expect(await orders.version('o1')).toBe(3)
expect(await orders.child('o1').version()).toBe(3)
expect(await orders.version('nope')).toBe(0)
})
it('stores one key per commit under <stream>/~e/<seq:12>', async () => {
const { events, driver } = setup()
await events.child('orders').append('o1', ['a'])
await events.child('orders').append('o1', ['b', 'c'])
expect(await collect(driver.inner.scan({ values: true }))).toEqual([
['orders/o1/~e/000000000001', { firstVersion: 1, at: 1000, events: ['a'] }],
['orders/o1/~e/000000000002', { firstVersion: 2, at: 1001, events: ['b', 'c'] }],
])
})
it('enforces expectedVersion', async () => {
const { events } = setup()
const s = events.child('s')
await s.append(['a'], { expectedVersion: 0 })
await expect(s.append(['b'], { expectedVersion: 0 })).rejects.toBeInstanceOf(ConcurrencyError)
await expect(s.append(['b'], { expectedVersion: 5 })).rejects.toMatchObject({ expectedVersion: 5, actualVersion: 1 })
expect(await s.append(['b'], { expectedVersion: 1 })).toEqual({ version: 2, commit: 2 })
await expect(s.append([], { expectedVersion: 1 })).rejects.toBeInstanceOf(ConcurrencyError)
expect(await s.append([])).toEqual({ version: 2, commit: 2 })
})
it('applies the arity rules', async () => {
const { events } = setup()
const orders = events.child('orders')
const id: string | undefined = undefined
await expect(orders.append(id as never, ['x'] as never)).rejects.toThrow(TypeError)
await expect(orders.version(id as never)).rejects.toThrow(TypeError)
await expect(collect(orders.read(id as never))).rejects.toThrow(TypeError)
await expect(events.append(['x'])).rejects.toThrow(/root/)
await expect(orders.append('o1', 'x' as never)).rejects.toThrow(/array/)
})
it('lists child streams and keeps nested streams separate', async () => {
const { events } = setup()
const orders = events.child('orders')
await orders.append('o1', ['a'])
await orders.append('o2', ['b'])
await orders.child('o1').child('shipments').append('s1', ['ship'])
await orders.child('o1').append('notes', ['n'])
expect(await collect(orders.keys())).toEqual(['o1', 'o2'])
expect(await collect(orders.child('o1').keys())).toEqual(['notes', 'shipments'])
expect((await collect(orders.read('o1'))).map((e) => e.data)).toEqual(['a'])
})
})
describe('read windows (property)', () => {
it('matches brute force for any commit layout', async () => {
await fc.assert(
fc.asyncProperty(
fc.array(fc.integer({ min: 1, max: 4 }), { maxLength: 25 }),
fc.record({
from: fc.option(fc.integer({ min: -1, max: 70 }), { nil: undefined }),
to: fc.option(fc.integer({ min: 0, max: 70 }), { nil: undefined }),
reverse: fc.boolean(),
limit: fc.option(fc.integer({ min: 0, max: 10 }), { nil: undefined }),
}),
async (sizes, opts) => {
const { events } = setup()
const s = events.child('s')
const all: number[] = []
for (const n of sizes) {
const batch = Array.from({ length: n }, (_, i) => all.length + i + 1)
await s.append(batch)
all.push(...batch)
}
let exp = all.filter((v) => v >= Math.max(1, opts.from ?? 1) && v <= (opts.to ?? Infinity))
if (opts.reverse) exp.reverse()
if (opts.limit !== undefined) exp = exp.slice(0, opts.limit)
const got = await collect(s.read(opts))
expect(got.map((e) => e.data)).toEqual(exp)
expect(got.map((e) => e.version)).toEqual(exp)
},
),
{ numRuns: 300 },
)
})
})
describe('costs', () => {
it('cached head: a warm append is a single setIfAbsent', async () => {
const { events, driver } = setup()
const s = events.child('s')
await s.append(['a'])
driver.reset()
await s.append(['b'])
await s.append(['c'], { expectedVersion: 2 })
expect(driver.calls).toEqual({ setIfAbsent: 2 })
})
it('cold head lookup is one reverse scan with limit 1', async () => {
const inner = new InMemoryKVDriver()
await setup([], inner).events.child('s').append(['a', 'b'])
const { events, driver } = setup([], inner)
expect(await events.child('s').append(['c'])).toEqual({ version: 3, commit: 2 })
expect(driver.calls).toEqual({ scan: 1, setIfAbsent: 1 })
})
it('read({ from }) finds single-event commits in one probe', async () => {
const { events, driver } = setup()
const s = events.child('s')
for (let i = 0; i < 200; i++) await s.append([i])
driver.reset()
expect((await collect(s.read({ from: 150, limit: 1 }))).map((e) => e.version)).toEqual([150])
expect(driver.calls).toEqual({ getMany: 1, scan: 1 })
})
it('read({ from }) uses O(log commits) probes for multi-event commits', async () => {
const { events, driver } = setup()
const s = events.child('s')
for (let i = 0; i < 1000; i++) await s.append(i % 3 === 0 ? [i, i] : [i])
driver.reset()
const [first] = await collect(s.read({ from: 777, limit: 1 }))
expect(first?.version).toBe(777)
// 1 guess probe + ceil(log8(1000)) search rounds, then the read scan. The guess hit an
// existing commit, so no head scan was needed.
expect(driver.calls.getMany).toBeLessThanOrEqual(1 + 4)
expect(driver.calls.scan).toBe(1)
})
})
describe('concurrency', () => {
it('a stale writer collides with the existing commit key', async () => {
const inner = new InMemoryKVDriver()
const a = setup([], inner).events.child('s')
const b = setup([], inner).events.child('s')
await a.append(['a1'])
await b.append(['b1']) // a's cached head is now stale
await expect(a.append(['a2'], { expectedVersion: 1 })).rejects.toMatchObject({ actualVersion: 2 })
// 'any' retries after refreshing the head
expect(await a.append(['a3'])).toEqual({ version: 3, commit: 3 })
expect((await collect(b.read())).map((e) => e.data)).toEqual(['a1', 'b1', 'a3'])
})
it('a stale cached head is refreshed before rejecting expectedVersion', async () => {
const inner = new InMemoryKVDriver()
const a = setup([], inner).events.child('s')
const b = setup([], inner).events.child('s')
await a.append(['a1'])
await b.append(['b1'])
expect(await a.append(['a2'], { expectedVersion: 2 })).toEqual({ version: 3, commit: 3 })
})
it('interleaved writers from two instances produce a dense, non-overlapping log', async () => {
const inner = new InMemoryKVDriver()
const a = setup([], inner).events.child('s')
const b = setup([], inner).events.child('s')
await Promise.all([a.append(['a1']), b.append(['b1']), a.append(['a2', 'a3']), b.append(['b2'])])
const log = await collect(a.read())
expect(log.map((e) => e.version)).toEqual([1, 2, 3, 4, 5])
expect(new Set(log.map((e) => e.data))).toEqual(new Set(['a1', 'a2', 'a3', 'b1', 'b2']))
})
it('exactly one of several optimistic writers at the same version wins', async () => {
const inner = new InMemoryKVDriver()
const writers = [0, 1, 2, 3].map(() => setup([], inner).events.child('s'))
const results = await Promise.allSettled(writers.map((w, i) => w.append([i], { expectedVersion: 0 })))
expect(results.filter((r) => r.status === 'fulfilled')).toHaveLength(1)
expect(results.filter((r) => r.status === 'rejected').every((r) => (r as PromiseRejectedResult).reason instanceof ConcurrencyError)).toBe(true)
})
it('without setIfAbsent, an in-process mutex serialises appends (single-writer mode)', async () => {
const { events, driver } = setup(['setIfAbsent'])
const s = events.child('s')
await Promise.all(Array.from({ length: 20 }, (_, i) => s.append([i])))
expect((await collect(s.read())).map((e) => e.version)).toEqual(Array.from({ length: 20 }, (_, i) => i + 1))
expect(driver.calls.setIfAbsent).toBeUndefined()
await expect(Promise.all([s.append(['x'], { expectedVersion: 20 }), s.append(['y'], { expectedVersion: 20 })])).rejects.toBeInstanceOf(ConcurrencyError)
})
})
test/helpers.ts
+28 -66
View File
@@ -1,83 +1,45 @@
import { BaseKVDriver, type BatchOp, type KVDriver, type ScanOptions } from '../src/kv/driver.js'
import { InMemoryKVDriver } from '../src/kv/memory.js'
import { BaseKVStorage, type ScanOptions } from '../src/kv-storage/base-kv-storage.js'
import { KVStorageInMemory } from '../src/kv-storage/kv-storage-in-memory.js'
/** A minimal driver implementing only the four required methods, over a plain Map. */
export class MinimalDriver extends BaseKVDriver {
readonly map = new Map<string, string>()
async getMany(keys: string[]) {
return keys.map((k) => (this.map.has(k) ? JSON.parse(this.map.get(k)!) : undefined))
}
async setMany(entries: [string, unknown][]) {
for (const [k, v] of entries) this.map.set(k, JSON.stringify(v))
}
async deleteMany(keys: string[]) {
for (const k of keys) this.map.delete(k)
}
async *scan({ gte, lt, limit = Infinity, reverse, values }: ScanOptions): AsyncGenerator<[string, unknown]> {
let keys = [...this.map.keys()].filter((k) => (gte === undefined || k >= gte) && (lt === undefined || k < lt)).sort()
if (reverse) keys.reverse()
keys = keys.slice(0, limit)
for (const k of keys) {
if (!this.map.has(k)) continue
yield [k, values ? JSON.parse(this.map.get(k)!) : undefined]
}
}
export async function collect<T>(iterable: AsyncIterable<T>): Promise<T[]> {
const out: T[] = []
for await (const item of iterable) out.push(item)
return out
}
export type Capability = 'setIfAbsent' | 'batch' | 'deleteRange'
/**
* Wraps a driver, counting calls per method and optionally hiding optional capabilities.
* `scan` counts each scan started (i.e. each seek).
* Wraps another KV storage and counts primitive calls. Also shows the wrapper pattern
* (e.g. an encryption layer): extend BaseKVStorage and delegate the five primitives.
*/
export class CountingDriver implements KVDriver {
export class KVStorageCounting extends BaseKVStorage {
calls: Record<string, number> = {}
setIfAbsent?: KVDriver['setIfAbsent']
batch?: KVDriver['batch']
deleteRange?: KVDriver['deleteRange']
constructor(
readonly inner: KVDriver = new InMemoryKVDriver(),
hide: Capability[] = [],
) {
if (inner.setIfAbsent && !hide.includes('setIfAbsent')) this.setIfAbsent = (e: [string, unknown][]) => (this.#count('setIfAbsent'), inner.setIfAbsent!(e))
if (inner.batch && !hide.includes('batch')) this.batch = (ops: BatchOp[]) => (this.#count('batch'), inner.batch!(ops))
if (inner.deleteRange && !hide.includes('deleteRange'))
this.deleteRange = (gte: string, lt: string) => (this.#count('deleteRange'), inner.deleteRange!(gte, lt))
}
#count(m: string): void {
this.calls[m] = (this.calls[m] ?? 0) + 1
constructor(readonly inner: BaseKVStorage = new KVStorageInMemory()) {
super()
}
reset(): void {
this.calls = {}
}
get total(): number {
return Object.values(this.calls).reduce((a, b) => a + b, 0)
readMany(keys: string[]) {
return this.count('readMany', () => this.inner.readMany(keys))
}
writeMany(entries: [string, unknown][]) {
return this.count('writeMany', () => this.inner.writeMany(entries))
}
removeMany(keys: string[]) {
return this.count('removeMany', () => this.inner.removeMany(keys))
}
writeIfAbsent(entries: [string, unknown][]) {
return this.count('writeIfAbsent', () => this.inner.writeIfAbsent(entries))
}
scan(options?: ScanOptions) {
return this.count('scan', () => this.inner.scan(options))
}
getMany(keys: string[]) {
this.#count('getMany')
return this.inner.getMany(keys)
}
setMany(entries: [string, unknown][]) {
this.#count('setMany')
return this.inner.setMany(entries)
}
deleteMany(keys: string[]) {
this.#count('deleteMany')
return this.inner.deleteMany(keys)
}
scan(opts: ScanOptions) {
this.#count('scan')
return this.inner.scan(opts)
private count<T>(name: string, fn: () => T): T {
this.calls[name] = (this.calls[name] ?? 0) + 1
return fn()
}
}
export async function collect<T>(it: AsyncIterable<T>): Promise<T[]> {
const out: T[] = []
for await (const x of it) out.push(x)
return out
}
test/http/http.test.ts
-92
View File
@@ -1,92 +0,0 @@
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { InMemoryKVDriver } from '../../src/kv/memory.js'
import type { KVDriver } from '../../src/kv/driver.js'
import { KVStorage } from '../../src/kv/node.js'
import { KVEventStorage } from '../../src/events/kv.js'
import { EventSourcedDocumentStorage } from '../../src/documents/event-sourced.js'
import { createKVHttpHandler } from '../../src/http/handler.js'
import { HttpKVDriver } from '../../src/http/client.js'
import { runKVDriverConformance } from '../../src/testing/kv-conformance.js'
import { collect, CountingDriver, MinimalDriver } from '../helpers.js'
import { serve } from './node-server.js'
/** A client wired straight to a handler, no socket. */
function inProcess(server: KVDriver = new InMemoryKVDriver(), caps?: ConstructorParameters<typeof HttpKVDriver>[0]['capabilities']) {
const handler = createKVHttpHandler(server, { basePath: '/kv' })
return new HttpKVDriver({ url: 'http://test/kv', capabilities: caps, fetch: async (input, init) => handler(new Request(input, init)) })
}
runKVDriverConformance('HttpKVDriver (in-process)', () => inProcess())
runKVDriverConformance('HttpKVDriver (minimal server driver, discovered capabilities)', () => {
const handler = createKVHttpHandler(new MinimalDriver())
return HttpKVDriver.connect({ url: 'http://test', fetch: async (input, init) => handler(new Request(input, init)) })
})
describe('over a real socket', () => {
let server: Awaited<ReturnType<typeof serve>>
const backends: InMemoryKVDriver[] = []
beforeAll(async () => {
// One server; each driver gets a fresh backend selected by path.
server = await serve(async (req) => {
const m = /^\/db(\d+)/.exec(new URL(req.url).pathname)
const backend = backends[Number(m?.[1])]
if (!backend) return new Response('no db', { status: 404 })
return createKVHttpHandler(backend, { basePath: `/db${m![1]}` })(req)
})
})
afterAll(() => server.close())
const fresh = () => {
backends.push(new InMemoryKVDriver())
return HttpKVDriver.connect({ url: `${server.url}/db${backends.length - 1}` })
}
runKVDriverConformance('HttpKVDriver (node:http)', fresh)
it('cancels a server-side scan when the client stops early', async () => {
const d = await fresh()
await d.setMany(Array.from({ length: 5000 }, (_, i) => [`k${String(i).padStart(5, '0')}`, i] as [string, number]))
const first: string[] = []
for await (const [k] of d.scan({})) {
first.push(k)
if (first.length === 3) break
}
expect(first).toEqual(['k00000', 'k00001', 'k00002'])
})
it('runs the whole stack (KV indexes, events, documents) over HTTP', async () => {
const kv = new KVStorage(await fresh(), { indexes: { users: { byAge: (u: { age: number }) => u.age } } })
await kv.child('users').set('1', { age: 30 })
expect(await collect(kv.child('users').findBy('byAge', { gte: 18 }))).toEqual([['1', { age: 30 }]])
const docs = new EventSourcedDocumentStorage(new KVEventStorage(kv), { indexes: { people: [{ fields: ['email'], unique: true }] } })
const id = await docs.child('people').insert({ email: 'a@x', n: 1 })
await docs.child('people').update(id, { $inc: { n: 1 } })
await expect(docs.child('people').insert({ email: 'a@x' })).rejects.toThrow(/Unique/)
expect(await docs.child('people').findOne({ email: 'a@x' })).toEqual({ key: id, data: { email: 'a@x', n: 2 } })
})
})
describe('handler', () => {
it('reports capabilities and rejects unsupported operations with 501', async () => {
const handler = createKVHttpHandler(new MinimalDriver())
const caps = await (await handler(new Request('http://x/capabilities'))).json()
expect(caps).toEqual({ setIfAbsent: false, batch: false, deleteRange: true })
const res = await handler(new Request('http://x/batch', { method: 'POST', body: JSON.stringify({ ops: [] }) }))
expect(res.status).toBe(501)
})
it('maps driver errors to 500 and unknown routes to 404', async () => {
const broken = new CountingDriver(new InMemoryKVDriver())
broken.getMany = () => Promise.reject(new Error('boom'))
const handler = createKVHttpHandler(broken)
expect((await handler(new Request('http://x/getMany', { method: 'POST', body: '{"keys":["a"]}' }))).status).toBe(500)
expect((await handler(new Request('http://x/nope'))).status).toBe(404)
await expect(inProcess(new MinimalDriver(), { batch: true }).batch!([])).rejects.toThrow(/HTTP 501/)
})
it('distinguishes null values from missing keys', async () => {
const d = inProcess()
await d.setMany([['n', null]])
expect(await d.getMany(['n', 'missing'])).toEqual([null, undefined])
})
})
test/http/node-server.ts
-36
View File
@@ -1,36 +0,0 @@
import { createServer, type IncomingHttpHeaders } from 'node:http'
import type { AddressInfo } from 'node:net'
/** Serves a `Request → Response` handler over a real socket (test helper). */
export async function serve(handler: (req: Request) => Promise<Response>): Promise<{ url: string; close: () => Promise<void> }> {
const server = createServer(async (req, res) => {
const chunks: Buffer[] = []
for await (const c of req) chunks.push(c as Buffer)
const headers = Object.entries(req.headers as IncomingHttpHeaders).flatMap(([k, v]) => (v === undefined ? [] : [[k, String(v)] as [string, string]]))
const request = new Request(`http://localhost${req.url}`, {
method: req.method,
headers,
body: req.method === 'GET' || req.method === 'HEAD' ? undefined : Buffer.concat(chunks),
})
const response = await handler(request)
res.writeHead(response.status, Object.fromEntries(response.headers))
if (!response.body) return void res.end()
const reader = response.body.getReader()
res.on('close', () => void reader.cancel().catch(() => {}))
try {
for (;;) {
const { done, value } = await reader.read()
if (done) break
res.write(value)
}
} finally {
res.end()
}
})
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve))
const { port } = server.address() as AddressInfo
return {
url: `http://127.0.0.1:${port}`,
close: () => new Promise((resolve) => (server.closeAllConnections(), server.close(() => resolve()))),
}
}
test/kv-storage/kv-storage-http.test.ts
+54
View File
@@ -0,0 +1,54 @@
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { KVStorageHttp } from '../../src/kv-storage/kv-storage-http.js'
import { KVStorageHttpServer } from '../../src/kv-storage/kv-storage-http-server.js'
import { KVStorageInMemory } from '../../src/kv-storage/kv-storage-in-memory.js'
import { EventStorageKV } from '../../src/event-storage/event-storage-kv.js'
import { DocumentStorageKV } from '../../src/document-storage/document-storage-kv.js'
import { EventSourcedDocumentStorage } from '../../src/document-storage/event-sourced-document-storage.js'
import { runKVStorageConformance } from './kv-storage.conformance.js'
import { serve } from './node-server.js'
runKVStorageConformance('KVStorageHttp (in-process)', () => {
const server = new KVStorageHttpServer(new KVStorageInMemory(), '/kv')
return new KVStorageHttp({ url: 'http://test/kv', fetch: async (input, init) => server.handle(new Request(input, init)) })
})
describe('over a real socket', () => {
let server: Awaited<ReturnType<typeof serve>>
const stores: KVStorageHttpServer[] = []
beforeAll(async () => {
// One socket; each test storage gets its own backend under /<n>.
server = await serve((request) => stores[Number(new URL(request.url).pathname.split('/')[1])]!.handle(request))
})
afterAll(() => server.close())
const fresh = () => {
stores.push(new KVStorageHttpServer(new KVStorageInMemory(), `/${stores.length}`))
return new KVStorageHttp({ url: `${server.url}/${stores.length - 1}` })
}
runKVStorageConformance('KVStorageHttp (node:http)', fresh)
it('stops a long scan early', async () => {
const db = fresh()
await db.writeMany(Array.from({ length: 5000 }, (_, i) => [`k${String(i).padStart(5, '0')}`, i] as [string, number]))
const seen: string[] = []
for await (const [key] of db.scan()) if (seen.push(key) === 3) break
expect(seen).toEqual(['k00000', 'k00001', 'k00002'])
})
it('runs events and event-sourced documents over HTTP', async () => {
const kv = fresh()
const docs = new EventSourcedDocumentStorage(new EventStorageKV(kv), new DocumentStorageKV(kv.child('projections')))
const id = await docs.child('people').insert({ email: 'a@x', n: 1 })
await docs.child('people').update(id, { $inc: { n: 1 } })
expect(await docs.child('people').findOne({ email: 'a@x' })).toEqual({ key: id, data: { email: 'a@x', n: 2 } })
})
})
describe('server', () => {
it('maps errors to 404 / 500', async () => {
const server = new KVStorageHttpServer(new KVStorageInMemory())
expect((await server.handle(new Request('http://x/nope', { method: 'POST', body: '{}' }))).status).toBe(404)
expect((await server.handle(new Request('http://x/readMany', { method: 'POST', body: '{"keys": 5}' }))).status).toBe(500)
})
})
test/kv-storage/kv-storage-in-memory.test.ts
+78
View File
@@ -0,0 +1,78 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { KVStorageInMemory } from '../../src/kv-storage/kv-storage-in-memory.js'
import type { BaseKVStorage } from '../../src/kv-storage/base-kv-storage.js'
import { collect, KVStorageCounting } from '../helpers.js'
import { runKVStorageConformance } from './kv-storage.conformance.js'
runKVStorageConformance('KVStorageInMemory', () => new KVStorageInMemory())
runKVStorageConformance('wrapper (KVStorageCounting)', () => new KVStorageCounting())
describe('costs', () => {
it('single-key operations are one primitive call', async () => {
const db = new KVStorageCounting()
const users = db.child('users')
await users.set('1', 'x')
await users.get('1')
await users.child('1').get()
await users.delete('1')
expect(db.calls).toEqual({ writeMany: 1, readMany: 2, removeMany: 1 })
})
it('keys() seeks once per child with descendants, not once per key', async () => {
const db = new KVStorageCounting()
const writes: [string, number][] = []
for (let c = 0; c < 50; c++) for (let d = 0; d < 20; d++) writes.push([`n/c${c}/d${d}`, d])
for (let c = 0; c < 50; c++) writes.push([`n/leaf${c}`, c])
await db.writeMany(writes)
db.reset()
expect(await collect(db.child('n').keys())).toHaveLength(100)
expect(db.calls).toEqual({ scan: 51 })
})
})
describe('model-based: random writes vs a reference map', () => {
const segment = fc.constantFrom('a', 'a-1', 'a.b', 'b', 'x/y', '%', '~e', '1', '10', '9', 'ab', 'a!', 'é')
const path = fc.array(segment, { minLength: 1, maxLength: 3 })
const op = fc.oneof(
fc.record({ t: fc.constant('set' as const), path, value: fc.integer() }),
fc.record({ t: fc.constant('delete' as const), path }),
fc.record({ t: fc.constant('deleteTree' as const), path }),
)
const at = (db: BaseKVStorage, p: string[]) => p.reduce<BaseKVStorage>((n, s) => n.child(s), db)
it('get, keys and entries agree with the model', async () => {
await fc.assert(
fc.asyncProperty(fc.array(op, { maxLength: 30 }), fc.array(segment, { maxLength: 2 }), async (ops, parent) => {
const db = new KVStorageInMemory()
const model = new Map<string, number>()
const isUnder = (p: string[], q: string[]) => q.length >= p.length && p.every((s, i) => q[i] === s)
for (const o of ops) {
if (o.t === 'set') (await at(db, o.path).set(o.value), model.set(JSON.stringify(o.path), o.value))
else if (o.t === 'delete') (await at(db, o.path).delete(), model.delete(JSON.stringify(o.path)))
else {
await at(db, o.path).deleteTree()
for (const k of [...model.keys()]) if (isUnder(o.path, JSON.parse(k))) model.delete(k)
}
}
for (const [k, v] of model) expect(await at(db, JSON.parse(k)).get()).toBe(v)
const children = new Map<string, number | undefined>()
for (const [k, v] of model) {
const p = JSON.parse(k) as string[]
if (p.length <= parent.length || !isUnder(parent, p)) continue
const own = p.length === parent.length + 1
if (own || !children.has(p[parent.length]!)) children.set(p[parent.length]!, own ? v : children.get(p[parent.length]!))
}
const node = at(db, parent)
const listed = await collect(node.keys())
expect([...listed].sort()).toEqual([...children.keys()].sort())
expect(await collect(node.keys({ reverse: true }))).toEqual([...listed].reverse())
const withValues = listed.filter((k) => children.get(k) !== undefined)
expect(await collect(node.entries())).toEqual(withValues.map((k) => [k, children.get(k)]))
expect(await collect(node.entries({ reverse: true }))).toEqual(withValues.reverse().map((k) => [k, children.get(k)]))
for (let i = 0; i < listed.length; i++) expect(await collect(node.keys({ after: listed[i]! }))).toEqual(listed.slice(i + 1))
}),
{ numRuns: 300 },
)
})
})
test/kv-storage/kv-storage.conformance.ts
+168
View File
@@ -0,0 +1,168 @@
import { beforeEach, describe, expect, it } from 'vitest'
import type { BaseKVStorage, ScanOptions } from '../../src/kv-storage/base-kv-storage.js'
import { InvalidKeyError } from '../../src/utils/errors.js'
import { collect } from '../helpers.js'
/** Behaviour every KV storage must have. `make` returns an empty storage. */
export function runKVStorageConformance(name: string, make: () => BaseKVStorage | Promise<BaseKVStorage>): void {
describe(`KV storage conformance: ${name}`, () => {
let db: BaseKVStorage
beforeEach(async () => {
db = await make()
})
const scanKeys = async (options: ScanOptions) => (await collect(db.scan(options))).map(([k]) => k)
describe('primitives', () => {
beforeEach(async () => {
await db.writeMany([
['b', 2],
['a/2', { n: 'a2' }],
['c', [3]],
['a', 1],
['a/1', 'a1'],
['a0', null],
['10', 10],
['9', 9],
])
})
it('reads in request order with undefined for missing keys and null preserved', async () => {
expect(await db.readMany(['c', 'missing', 'a/2', 'a0'])).toEqual([[3], undefined, { n: 'a2' }, null])
})
it('scans in key order with bounds, reverse, limit and optional values', async () => {
expect(await scanKeys({})).toEqual(['10', '9', 'a', 'a/1', 'a/2', 'a0', 'b', 'c'])
expect(await scanKeys({ gte: 'a/', lt: 'a0' })).toEqual(['a/1', 'a/2'])
expect(await scanKeys({ reverse: true, gte: 'a', lt: 'b' })).toEqual(['a0', 'a/2', 'a/1', 'a'])
expect(await scanKeys({ limit: 2, reverse: true })).toEqual(['c', 'b'])
expect(await scanKeys({ limit: 0 })).toEqual([])
expect(await collect(db.scan({ gte: 'a/', lt: 'a0', values: true }))).toEqual([
['a/1', 'a1'],
['a/2', { n: 'a2' }],
])
expect((await collect(db.scan({ gte: 'b' })))[0]).toEqual(['b', undefined])
})
it('removes keys', async () => {
await db.removeMany(['b', 'missing'])
expect(await db.readMany(['b', 'c'])).toEqual([undefined, [3]])
})
it('writeIfAbsent writes only absent keys, and only one concurrent claim wins', async () => {
expect(
await db.writeIfAbsent([
['a', 'x'],
['new', 'y'],
]),
).toEqual([false, true])
expect(await db.readMany(['a', 'new'])).toEqual([1, 'y'])
const results = await Promise.all([1, 2, 3].map((n) => db.writeIfAbsent([['claim', n]])))
expect(results.flat().filter(Boolean)).toHaveLength(1)
})
it('keeps scanning correctly when keys change mid-scan', async () => {
// Keys removed mid-scan may or may not still appear (stores may read ahead), but every key
// present throughout must appear exactly once, in order.
const seen: string[] = []
for await (const [key] of db.scan({})) {
seen.push(key)
if (key === 'a') await db.removeMany(['a', 'a/1'])
}
expect(seen.filter((k) => k !== 'a/1')).toEqual(['10', '9', 'a', 'a/2', 'a0', 'b', 'c'])
})
it('does not share references with callers', async () => {
const [value] = (await db.readMany(['a/2'])) as [{ n: string }]
value.n = 'changed'
expect(await db.readMany(['a/2'])).toEqual([{ n: 'a2' }])
})
})
describe('nodes', () => {
it('reads and writes own and child values at the same key', async () => {
const users = db.child('users')
await users.set('123', { name: 'Ada' })
expect(await users.child('123').get()).toEqual({ name: 'Ada' })
await users.child('123').set({ name: 'Bob' })
expect(await users.get('123')).toEqual({ name: 'Bob' })
expect(await db.readMany(['users/123'])).toEqual([{ name: 'Bob' }])
await db.child('n').set(5, 'five')
expect(await db.child('n').get('5')).toBe('five')
})
it('batch reads, writes and deletes children', async () => {
const n = db.child('n')
await n.setMany([
['a', 1],
['b', 2],
])
expect(await n.getMany(['b', 'x', 'a'])).toEqual([2, undefined, 1])
await n.deleteMany(['a'])
expect(await n.getMany(['a', 'b'])).toEqual([undefined, 2])
})
it('keeps own values and children independent', async () => {
const user = db.child('users').child('1')
await user.set('own')
await user.set('name', 'Ada')
await user.delete()
expect(await user.get()).toBeUndefined()
expect(await user.get('name')).toBe('Ada')
})
it('rejects passed-but-undefined keys, root values and undefined values', async () => {
const users = db.child('users')
const id: string | undefined = undefined
await expect(users.get(id as never)).rejects.toThrow(InvalidKeyError)
await expect(users.delete(null as never)).rejects.toThrow(TypeError)
await expect(users.set(id as never, 1 as never)).rejects.toThrow(TypeError)
await expect(users.set('x', undefined)).rejects.toThrow(/undefined/)
await expect(db.get()).rejects.toThrow(/root/)
expect(() => db.child('')).toThrow(InvalidKeyError)
})
it('escapes keys so special characters round-trip', async () => {
const n = db.child('a/b')
for (const key of ['x/y', '50%', '~e', 'a-1', 'a.b', 'a b', 'é']) await n.set(key, key)
for (const key of ['x/y', '50%', '~e', 'a-1', 'a.b', 'a b', 'é']) expect(await n.get(key)).toBe(key)
expect((await collect(n.keys())).sort()).toEqual(['50%', 'a b', 'a-1', 'a.b', 'x/y', '~e', 'é'].sort())
})
it('lists direct children (including ones with only descendants), skipping reserved keys', async () => {
const n = db.child('n')
await n.set('a', 1)
await n.child('b').set('deep', 1)
await n.set('c', 3)
await n.child('c').child('x').set('y', 1)
await n.set('a-1', 4)
await db.writeMany([['n/~e/000000000001', {}]])
expect(await collect(n.keys())).toEqual(['a', 'a-1', 'b', 'c'])
expect(await collect(n.keys({ reverse: true }))).toEqual(['c', 'b', 'a-1', 'a'])
expect(await collect(n.keys({ after: 'a', limit: 2 }))).toEqual(['a-1', 'b'])
expect(await collect(n.keys({ after: 'c', reverse: true }))).toEqual(['b', 'a-1', 'a'])
expect(await collect(n.entries())).toEqual([
['a', 1],
['a-1', 4],
['c', 3],
])
expect(await collect(n.entries({ reverse: true, limit: 2 }))).toEqual([
['c', 3],
['a-1', 4],
])
expect(await collect(db.keys())).toEqual(['n'])
})
it('deleteTree removes a subtree but not lookalike siblings', async () => {
const n = db.child('n')
await n.set('a', 1)
await n.child('a').child('b').set('c', 3)
await n.set('a!', 2)
await n.set('a0', 3)
await n.deleteTree('a')
expect(await collect(n.keys())).toEqual(['a0', 'a!']) // 'a!' is stored as 'a~21'
await db.deleteTree()
expect(await collect(db.scan())).toEqual([])
})
})
})
}
test/kv-storage/node-server.ts
+24
View File
@@ -0,0 +1,24 @@
import { createServer } from 'node:http'
import type { AddressInfo } from 'node:net'
/** Serves a `Request → Response` handler over a real socket (test helper). */
export async function serve(handle: (request: Request) => Promise<Response>): Promise<{ url: string; close: () => Promise<void> }> {
const server = createServer(async (req, res) => {
const chunks: Buffer[] = []
for await (const chunk of req) chunks.push(chunk as Buffer)
const response = await handle(
new Request(`http://localhost${req.url}`, { method: req.method, body: req.method === 'POST' ? Buffer.concat(chunks) : undefined }),
)
res.writeHead(response.status, Object.fromEntries(response.headers))
if (!response.body) return void res.end()
const reader = response.body.getReader()
res.on('close', () => void reader.cancel().catch(() => {}))
for (let next = await reader.read(); !next.done; next = await reader.read()) res.write(next.value)
res.end()
})
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve))
return {
url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
close: () => new Promise((resolve) => (server.closeAllConnections(), server.close(() => resolve()))),
}
}
test/kv/indexes.test.ts
-256
View File
@@ -1,256 +0,0 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { KVStorage, type KVStorageOptions } from '../../src/kv/node.js'
import { InMemoryKVDriver } from '../../src/kv/memory.js'
import type { KVDriver } from '../../src/kv/driver.js'
import { encodeIndexValue } from '../../src/core/encoding.js'
import { collect, CountingDriver, type Capability } from '../helpers.js'
type User = { email?: string; age?: number; tags?: string[] }
type Post = { createdAt: string; tags: string[] }
const indexes: KVStorageOptions['indexes'] = {
users: {
byEmail: (u: User) => u.email,
byAge: (u: User) => u.age,
byTag: (u: User) => u.tags,
},
'users/*/posts': { byDate: (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
}
function setup(hide: Capability[] = []) {
const driver = new CountingDriver(new InMemoryKVDriver(), hide)
return { driver, db: new KVStorage(driver, { indexes }) }
}
const modes: [string, Capability[]][] = [
['atomic batch', []],
['non-atomic', ['batch']],
]
describe.each(modes)('indexes (%s)', (_, hide) => {
it('writes entries under <node>/~i/<name>/<enc>/<child>', async () => {
const { db, driver } = setup(hide)
await db.child('users').set('123', { email: 'sada@x.io', age: 30 })
const keys = (await collect(driver.inner.scan({}))).map(([k]) => k)
expect(keys).toEqual([`users/123`, `users/~i/byAge/${encodeIndexValue(30)}/123`, `users/~i/byEmail/${encodeIndexValue('sada@x.io')}/123`])
})
it('finds by exact value and keeps entries in sync on update and delete', async () => {
const { db } = setup(hide)
const users = db.child('users')
await users.set('1', { email: 'a@x', age: 30 })
await users.set('2', { email: 'b@x', age: 30 })
expect(await collect(users.findBy('byEmail', 'a@x'))).toEqual([['1', { email: 'a@x', age: 30 }]])
expect((await collect(users.findBy('byAge', 30))).map(([k]) => k)).toEqual(['1', '2'])
await users.set('1', { email: 'c@x', age: 31 })
expect(await collect(users.findBy('byEmail', 'a@x'))).toEqual([])
expect((await collect(users.findBy('byEmail', 'c@x'))).map(([k]) => k)).toEqual(['1'])
await users.delete('2')
expect(await collect(users.findBy('byAge', 30))).toEqual([])
})
it('self-form writes on a child update the parent index', async () => {
const { db } = setup(hide)
const u = db.child('users').child('7')
await u.set({ email: 'self@x' })
expect((await collect(db.child('users').findBy('byEmail', 'self@x'))).map(([k]) => k)).toEqual(['7'])
await u.delete()
expect(await collect(db.child('users').findBy('byEmail', 'self@x'))).toEqual([])
})
it('deleteTree removes the child entry', async () => {
const { db, driver } = setup(hide)
await db.child('users').set('7', { email: 'e' })
await db.child('users').child('7').child('posts').set('p', { createdAt: '2024', tags: [] })
await db.child('users').deleteTree('7')
expect(await collect(driver.inner.scan({}))).toEqual([])
})
it('supports multi-entry values, de-duplicated in range queries', async () => {
const { db } = setup(hide)
const users = db.child('users')
await users.set('1', { tags: ['a', 'b', 'a'] })
await users.set('2', { tags: ['b'] })
expect((await collect(users.findBy('byTag', 'b'))).map(([k]) => k)).toEqual(['1', '2'])
expect((await collect(users.findBy('byTag', { gte: 'a', lt: 'c' }))).map(([k]) => k)).toEqual(['1', '2'])
await users.set('1', { tags: ['c'] })
expect((await collect(users.findBy('byTag', 'b'))).map(([k]) => k)).toEqual(['2'])
})
it('range queries are ordered by value and honour bounds, limit and reverse', async () => {
const { db } = setup(hide)
const users = db.child('users')
await users.setMany([
['a', { age: 10 }],
['b', { age: 9 }],
['c', { age: -1.5 }],
['d', { age: 100 }],
['e', { age: 20 }],
])
const ages = async (q: object, o = {}) => (await collect(users.findBy('byAge', q as never, o))).map(([, v]) => (v as User).age)
expect(await ages({})).toEqual([-1.5, 9, 10, 20, 100])
expect(await ages({ gte: 9, lt: 20 })).toEqual([9, 10])
expect(await ages({ gt: 9, lte: 20 })).toEqual([10, 20])
expect(await ages({ gte: 0 }, { limit: 2 })).toEqual([9, 10])
expect(await ages({ lt: 100 }, { reverse: true })).toEqual([20, 10, 9, -1.5])
})
it('applies pattern indexes with * to nested nodes only', async () => {
const { db } = setup(hide)
const posts = db.child('users').child('u1').child('posts')
await posts.set('p1', { createdAt: '2024-02', tags: ['x'] })
await posts.set('p2', { createdAt: '2024-01', tags: ['y'] })
expect((await collect(posts.findBy('byDate', { gte: '2024' }))).map(([k]) => k)).toEqual(['p2', 'p1'])
await expect(db.child('other').findBy('byDate', 'x').next()).rejects.toThrow(/No index/)
})
it('handles child keys and values needing escaping', async () => {
const { db } = setup(hide)
const users = db.child('users')
await users.set('a/b%~', { email: 'we/ird%~@x' })
expect(await collect(users.findBy('byEmail', 'we/ird%~@x'))).toEqual([['a/b%~', { email: 'we/ird%~@x' }]])
})
})
describe('index costs', () => {
it('non-indexed nodes do not read before writing', async () => {
const { db, driver } = setup()
driver.reset()
await db.child('other').set('1', { email: 'x' })
await db.child('other').setMany([['2', {}]])
expect(driver.calls).toEqual({ setMany: 2 })
})
it('indexed writes: one getMany then one atomic batch', async () => {
const { db, driver } = setup()
driver.reset()
await db.child('users').setMany([
['1', { email: 'a' }],
['2', { email: 'b' }],
])
expect(driver.calls).toEqual({ getMany: 1, batch: 1 })
})
it('findBy with a small limit makes one value read', async () => {
const { db, driver } = setup()
await db.child('users').setMany(Array.from({ length: 300 }, (_, i) => [String(i), { age: 1 }] as [string, User]))
driver.reset()
expect(await collect(db.child('users').findBy('byAge', 1, { limit: 1 }))).toHaveLength(1)
expect(driver.calls).toEqual({ scan: 1, getMany: 1 })
driver.reset()
expect(await collect(db.child('users').findBy('byAge', 1))).toHaveLength(300)
expect(driver.calls).toEqual({ scan: 1, getMany: 3 })
})
})
describe('non-atomic crash recovery (read-repair)', () => {
/** Fails the n-th mutating call. */
class CrashingDriver extends CountingDriver {
failAt = Infinity
#n = 0
override setMany(entries: [string, unknown][]) {
if (++this.#n === this.failAt) return Promise.reject(new Error('crash'))
return super.setMany(entries)
}
override deleteMany(keys: string[]) {
if (++this.#n === this.failAt) return Promise.reject(new Error('crash'))
return super.deleteMany(keys)
}
}
it('skips and deletes stale entries left by a crash after the index add', async () => {
const driver = new CrashingDriver(new InMemoryKVDriver(), ['batch'])
const users = new KVStorage(driver, { indexes }).child('users')
await users.set('1', { email: 'old' })
driver.failAt = 2 + 2 // adds succeed, value write crashes
await expect(users.set('1', { email: 'new' })).rejects.toThrow('crash')
// The 'new' entry is a dangling hint; the value still says 'old'.
expect(await collect(users.findBy('byEmail', 'new'))).toEqual([])
expect(await collect(users.findBy('byEmail', 'old'))).toEqual([['1', { email: 'old' }]])
const entries = (await collect(driver.inner.scan({ gte: 'users/~i/' }))).map(([k]) => k)
expect(entries).toEqual([`users/~i/byEmail/${encodeIndexValue('old')}/1`])
})
it('skips stale entries left by a crash before the index remove', async () => {
const driver = new CrashingDriver(new InMemoryKVDriver(), ['batch'])
const users = new KVStorage(driver, { indexes }).child('users')
await users.set('1', { email: 'old' })
driver.failAt = 2 + 3 // adds + value succeed, remove crashes
await expect(users.set('1', { email: 'new' })).rejects.toThrow('crash')
expect(await collect(users.findBy('byEmail', 'old'))).toEqual([])
expect(await collect(users.findBy('byEmail', 'new'))).toEqual([['1', { email: 'new' }]])
})
it('can leave stale entries in place with readRepair: false', async () => {
const driver = new InMemoryKVDriver()
await driver.setMany([[`users/~i/byEmail/${encodeIndexValue('x')}/1`, true]])
const users = new KVStorage(driver, { indexes, readRepair: false }).child('users')
expect(await collect(users.findBy('byEmail', 'x'))).toEqual([])
expect(await collect(driver.scan({}))).toHaveLength(1)
})
})
describe('index declarations', () => {
it('rejects the same index name on overlapping patterns', () => {
expect(() => new KVStorage(new InMemoryKVDriver(), { indexes: { 'a/*': { i: () => 1 }, '*/b': { i: () => 2 } } })).toThrow(/overlapping/)
expect(() => new KVStorage(new InMemoryKVDriver(), { indexes: { 'a/*': { i: () => 1 }, '*/b': { j: () => 2 } } })).not.toThrow()
})
it('rejects non-indexable index function results', async () => {
const db = new KVStorage(new InMemoryKVDriver(), { indexes: { '': { bad: () => ({}) as never } } })
await expect(db.set('x', 1)).rejects.toThrow(/non-indexable/)
})
})
describe('property: findBy equals brute-force filter', () => {
const val = fc.record({
age: fc.option(fc.integer({ min: -3, max: 3 }), { nil: undefined }),
tags: fc.array(fc.constantFrom('a', 'b', 'c/d'), { maxLength: 3 }),
})
const op = fc.oneof(
fc.record({ t: fc.constant('set' as const), k: fc.constantFrom('1', '2', '3', 'x/y'), v: val }),
fc.record({ t: fc.constant('del' as const), k: fc.constantFrom('1', '2', '3', 'x/y') }),
)
for (const [mode, hide] of modes) {
it(mode, async () => {
await fc.assert(
fc.asyncProperty(fc.array(op, { maxLength: 20 }), fc.integer({ min: -4, max: 4 }), fc.integer({ min: -4, max: 4 }), async (ops, lo, hi) => {
const driver: KVDriver = new CountingDriver(new InMemoryKVDriver(), hide)
const users = new KVStorage(driver, { indexes }).child('users')
const model = new Map<string, User>()
for (const o of ops) {
if (o.t === 'set') {
await users.set(o.k, o.v)
model.set(o.k, o.v)
} else {
await users.delete(o.k)
model.delete(o.k)
}
}
const got = (await collect(users.findBy('byAge', { gte: lo, lt: hi }))).map(([k]) => k).sort()
const exp = [...model]
.filter(([, v]) => v.age !== undefined && v.age >= lo && v.age < hi)
.map(([k]) => k)
.sort()
expect(got).toEqual(exp)
for (const tag of ['a', 'b', 'c/d']) {
const g = (await collect(users.findBy('byTag', tag))).map(([k]) => k).sort()
expect(g).toEqual(
[...model]
.filter(([, v]) => v.tags?.includes(tag))
.map(([k]) => k)
.sort(),
)
}
// No stale entries are ever left behind by clean writes.
let entryCount = 0
for await (const _ of driver.scan({ gte: 'users/~i/', lt: 'users/~i0' })) entryCount++
const expectedEntries = [...model.values()].reduce((n, v) => n + (v.age !== undefined ? 1 : 0) + new Set(v.tags).size, 0)
expect(entryCount).toBe(expectedEntries)
}),
{ numRuns: 200 },
)
})
}
})
test/kv/memory-driver.test.ts
-28
View File
@@ -1,28 +0,0 @@
import { describe, expect, it } from 'vitest'
import { InMemoryKVDriver } from '../../src/kv/memory.js'
import { runKVDriverConformance } from '../../src/testing/kv-conformance.js'
import { CountingDriver, MinimalDriver } from '../helpers.js'
runKVDriverConformance('InMemoryKVDriver', () => new InMemoryKVDriver())
runKVDriverConformance('BaseKVDriver (minimal)', () => new MinimalDriver())
runKVDriverConformance('CountingDriver(InMemory)', () => new CountingDriver())
describe('InMemoryKVDriver', () => {
it('keeps order after large bulk merges', async () => {
const d = new InMemoryKVDriver()
const keys = Array.from({ length: 5000 }, (_, i) => `k${(i * 7919) % 5000}`)
await d.setMany(keys.slice(0, 2500).map((k) => [k, 1]))
await d.setMany(keys.slice(2500).map((k) => [k, 2]))
const seen: string[] = []
for await (const [k] of d.scan({})) seen.push(k)
expect(seen).toEqual([...new Set(keys)].sort())
expect(d.size).toBe(5000)
})
it('can skip cloning', async () => {
const d = new InMemoryKVDriver({ clone: false })
const v = { n: 1 }
await d.setMany([['k', v]])
expect((await d.getMany(['k']))[0]).toBe(v)
})
})
test/kv/node.test.ts
-354
View File
@@ -1,354 +0,0 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { KVStorage } from '../../src/kv/node.js'
import { InMemoryKVDriver } from '../../src/kv/memory.js'
import { escapeSegment } from '../../src/core/path.js'
import { InvalidKeyError } from '../../src/errors.js'
import { collect, CountingDriver, MinimalDriver } from '../helpers.js'
function setup() {
const driver = new CountingDriver()
const db = new KVStorage(driver)
return { driver, db }
}
describe('KVNode basics', () => {
it('reads and writes own and child values at the same key', async () => {
const { db } = setup()
const users = db.child('users')
await users.set('123', { name: 'Ada' })
expect(await users.get('123')).toEqual({ name: 'Ada' })
expect(await users.child('123').get()).toEqual({ name: 'Ada' })
await users.child('123').set({ name: 'Bob' })
expect(await users.get('123')).toEqual({ name: 'Bob' })
expect(users.child('123').selfKey).toBe('users/123')
})
it('treats number keys as strings', async () => {
const { db } = setup()
await db.child('n').set(5, 'five')
expect(await db.child('n').get('5')).toBe('five')
})
it('escapes keys containing the separator, % and leading ~', async () => {
const { db, driver } = setup()
const n = db.child('a/b')
await n.set('~x', 1)
await n.set('50%', 2)
expect(await collect(driver.inner.scan({}))).toEqual([
['a%2Fb/%7Ex', undefined],
['a%2Fb/50%25', undefined],
])
// Ordered by escaped segment: '%7Ex' < '50%25'.
expect(await collect(n.keys())).toEqual(['~x', '50%'])
})
it('own value and children are independent keys', async () => {
const { db } = setup()
const u = db.child('users').child('1')
await u.set('own')
await u.set('name', 'Ada')
await u.delete()
expect(await u.get()).toBeUndefined()
expect(await u.get('name')).toBe('Ada')
})
it('getMany / setMany / deleteMany', async () => {
const { db, driver } = setup()
const n = db.child('n')
driver.reset()
await n.setMany([
['a', 1],
['b', 2],
])
expect(await n.getMany(['b', 'x', 'a'])).toEqual([2, undefined, 1])
await n.deleteMany(['a', 'x'])
expect(await n.getMany(['a', 'b'])).toEqual([undefined, 2])
expect(driver.calls).toEqual({ setMany: 1, getMany: 2, deleteMany: 1 })
})
it('single-key ops make exactly one driver call', async () => {
const { db, driver } = setup()
const n = db.child('users')
driver.reset()
await n.set('1', 'x')
await n.get('1')
await n.child('1').get()
await n.delete('1')
expect(driver.calls).toEqual({ setMany: 1, getMany: 2, deleteMany: 1 })
})
it('rejects setting undefined', async () => {
const { db } = setup()
await expect(db.child('a').set('b', undefined)).rejects.toThrow(TypeError)
})
})
describe('arity dispatch', () => {
it('throws on passed-but-undefined or null keys instead of reading self', async () => {
const { db } = setup()
const users = db.child('users')
const maybe: string | undefined = undefined
await expect(users.get(maybe as never)).rejects.toThrow(InvalidKeyError)
await expect(users.get(null as never)).rejects.toThrow(TypeError)
await expect(users.set(maybe as never, 1 as never)).rejects.toThrow(TypeError)
await expect(users.delete(maybe as never)).rejects.toThrow(TypeError)
await expect(users.getMany([maybe as never])).rejects.toThrow(TypeError)
await expect(users.deleteTree(maybe as never)).rejects.toThrow(TypeError)
expect(() => users.child(maybe as never)).toThrow(TypeError)
})
it('root has no own value', async () => {
const { db } = setup()
await expect(db.get()).rejects.toThrow(TypeError)
await expect(db.set(1)).rejects.toThrow(TypeError)
await expect(db.delete()).rejects.toThrow(TypeError)
})
it('rejects empty keys', () => {
const { db } = setup()
expect(() => db.child('')).toThrow(InvalidKeyError)
})
})
describe('keys / entries', () => {
it('lists direct children, including ones with only descendants, skipping reserved', async () => {
const { db } = setup()
const n = db.child('n')
await n.set('a', 1)
await n.child('b').set('deep', 1)
await n.set('c', 3)
await n.child('c').child('x').set('y', 1)
await db.driver.setMany([['n/~i/idx/v/a', true]])
await n.set('\u{1F600}', 'emoji')
expect(await collect(n.keys())).toEqual(['a', 'b', 'c', '\u{1F600}'])
expect(await collect(n.keys({ reverse: true }))).toEqual(['\u{1F600}', 'c', 'b', 'a'])
expect(await collect(n.keys({ after: 'a', limit: 2 }))).toEqual(['b', 'c'])
expect(await collect(n.keys({ after: 'c', reverse: true }))).toEqual(['b', 'a'])
expect(await collect(n.entries())).toEqual([
['a', 1],
['c', 3],
['\u{1F600}', 'emoji'],
])
expect(await collect(n.entries({ reverse: true, limit: 2 }))).toEqual([
['\u{1F600}', 'emoji'],
['c', 3],
])
})
it('root keys skip the reserved ~p/~e roots', async () => {
const { db } = setup()
await db.driver.setMany([
['~p/users/1', {}],
['users/1', {}],
])
expect(await collect(db.keys())).toEqual(['users'])
})
it('orders siblings that extend another with a below-separator char first', async () => {
const { db } = setup()
const n = db.child('n')
await n.set('a', 1)
await n.set('a-1', 2)
await n.child('a').set('x', 3)
expect(await collect(n.keys())).toEqual(['a-1', 'a'])
expect(await collect(n.keys({ reverse: true }))).toEqual(['a', 'a-1'])
expect(await collect(n.entries())).toEqual([
['a-1', 2],
['a', 1],
])
})
it('is a skip-scan: seeks are O(children), not O(subtree)', async () => {
const { db, driver } = setup()
const n = db.child('n')
const writes: [string, unknown][] = []
for (let c = 0; c < 50; c++) {
writes.push([`n/c${String(c).padStart(3, '0')}`, c])
for (let d = 0; d < 100; d++) writes.push([`n/c${String(c).padStart(3, '0')}/d${d}`, d])
}
await driver.inner.setMany(writes)
driver.reset()
const keys = await collect(n.keys())
expect(keys).toHaveLength(50)
// One initial scan plus one seek per child with descendants.
expect(driver.calls.scan).toBe(51)
expect(driver.calls.getMany).toBeUndefined()
})
it('siblings without descendants share one iterator', async () => {
const { db, driver } = setup()
const n = db.child('n')
await n.setMany(Array.from({ length: 100 }, (_, i) => [`k${i}`, i] as [string, number]))
driver.reset()
expect(await collect(n.keys())).toHaveLength(100)
expect(driver.calls.scan).toBe(1)
})
})
describe('deleteTree', () => {
it('deletes own value and all descendants but not lookalike siblings', async () => {
const { db } = setup()
const n = db.child('n')
await n.set('a', 1)
await n.child('a').set('b', 2)
await n.child('a').child('b').set('c', 3)
await n.set('a!', 'sibling')
await n.set('a0', 'sibling')
await n.deleteTree('a')
expect(await collect(n.keys())).toEqual(['a!', 'a0'])
await n.child('a0').deleteTree()
expect(await collect(n.keys())).toEqual(['a!'])
})
it('works without deleteRange, and on root', async () => {
const driver = new CountingDriver(new InMemoryKVDriver(), ['deleteRange'])
const db = new KVStorage(driver)
for (let i = 0; i < 2500; i++) await db.child('x').child(String(i)).set('v', i)
await db.child('y').set(1, 1)
await db.child('x').deleteTree()
expect(await collect(db.keys())).toEqual(['y'])
await db.deleteTree()
expect(await collect(driver.scan({}))).toEqual([])
})
})
describe('tree', () => {
it('assembles descendants up to depth, field-per-key style', async () => {
const { db } = setup()
const p = db.child('profile').child('9')
await p.set('username', 'ada')
await p.set('age', 36)
await p.child('address').set('city', 'London')
await p.child('address').child('geo').set('lat', 51)
await p.child('address').set('own')
await db.driver.setMany([['profile/9/~i/x/y', true]])
expect(await p.tree({ depth: 1 })).toEqual({ username: 'ada', age: 36, address: 'own' })
expect(await p.tree({ depth: 2 })).toEqual({ username: 'ada', age: 36, address: { '': 'own', city: 'London', geo: undefined } })
expect(await p.tree({ depth: 3 })).toEqual({ username: 'ada', age: 36, address: { '': 'own', city: 'London', geo: { lat: 51 } } })
})
})
// ------------------------------------------------------------------ model-based
type Path = string[]
const SEP = '/'
const segArb = fc.constantFrom('a', 'a-1', 'a.b', 'b', 'a/x', '%', '~t', '1', '10', '9', 'ab', 'a!', 'é')
const pathArb = fc.array(segArb, { minLength: 1, maxLength: 3 })
const valArb = fc.oneof(fc.integer(), fc.string(), fc.constant(null), fc.record({ n: fc.integer() }))
type Op =
| { t: 'set'; path: Path; value: unknown; selfForm: boolean }
| { t: 'delete'; path: Path; selfForm: boolean }
| { t: 'deleteTree'; path: Path }
| { t: 'setMany'; parent: Path; entries: [string, unknown][] }
const opArb: fc.Arbitrary<Op> = fc.oneof(
{ weight: 5, arbitrary: fc.record({ t: fc.constant('set' as const), path: pathArb, value: valArb, selfForm: fc.boolean() }) },
{ weight: 2, arbitrary: fc.record({ t: fc.constant('delete' as const), path: pathArb, selfForm: fc.boolean() }) },
{ weight: 1, arbitrary: fc.record({ t: fc.constant('deleteTree' as const), path: pathArb }) },
{
weight: 1,
arbitrary: fc.record({
t: fc.constant('setMany' as const),
parent: fc.array(segArb, { maxLength: 2 }),
entries: fc.array(fc.tuple(segArb, valArb), { maxLength: 4 }),
}),
},
)
const enc = (p: Path) => JSON.stringify(p)
const isPrefix = (p: Path, q: Path) => p.length <= q.length && p.every((s, i) => q[i] === s)
/** Expected child order: escaped segment + separator, lexicographic. */
function expectedChildren(model: Map<string, unknown>, parent: Path): { key: string; own: unknown; hasOwn: boolean }[] {
const kids = new Map<string, { own: unknown; hasOwn: boolean }>()
for (const [k, v] of model) {
const p = JSON.parse(k) as Path
if (p.length <= parent.length || !isPrefix(parent, p)) continue
const seg = p[parent.length]!
const e = kids.get(seg) ?? { own: undefined, hasOwn: false }
if (p.length === parent.length + 1) Object.assign(e, { own: v, hasOwn: true })
kids.set(seg, e)
}
return [...kids]
.map(([key, e]) => ({ key, ...e }))
.sort((x, y) => {
const a = escapeSegment(x.key, SEP) + SEP
const b = escapeSegment(y.key, SEP) + SEP
return a < b ? -1 : a > b ? 1 : 0
})
}
function nodeAt(db: KVStorage, path: Path) {
let n = db as ReturnType<KVStorage['child']>
for (const s of path) n = n.child(s)
return n
}
describe('model-based (random ops vs reference Map)', () => {
for (const [name, make] of [
['InMemory', () => new InMemoryKVDriver()],
['Minimal', () => new MinimalDriver()],
] as const) {
it(`matches the model (${name})`, async () => {
await fc.assert(
fc.asyncProperty(
fc.array(opArb, { maxLength: 30 }),
fc.array(fc.array(segArb, { maxLength: 2 }), { minLength: 1, maxLength: 3 }),
async (ops, probes) => {
const db = new KVStorage(make())
const model = new Map<string, unknown>()
for (const op of ops) {
switch (op.t) {
case 'set': {
if (op.selfForm) await nodeAt(db, op.path).set(op.value)
else await nodeAt(db, op.path.slice(0, -1)).set(op.path.at(-1)!, op.value)
model.set(enc(op.path), op.value)
break
}
case 'delete': {
if (op.selfForm) await nodeAt(db, op.path).delete()
else await nodeAt(db, op.path.slice(0, -1)).delete(op.path.at(-1)!)
model.delete(enc(op.path))
break
}
case 'deleteTree': {
await nodeAt(db, op.path).deleteTree()
for (const k of [...model.keys()]) if (isPrefix(op.path, JSON.parse(k))) model.delete(k)
break
}
case 'setMany': {
await nodeAt(db, op.parent).setMany(op.entries)
for (const [k, v] of op.entries) model.set(enc([...op.parent, k]), v)
break
}
}
}
for (const [k, v] of model) expect(await nodeAt(db, JSON.parse(k)).get()).toEqual(v)
for (const parent of probes) {
const node = nodeAt(db, parent)
const exp = expectedChildren(model, parent)
const keys = exp.map((e) => e.key)
expect(await collect(node.keys())).toEqual(keys)
expect(await collect(node.keys({ reverse: true }))).toEqual([...keys].reverse())
expect(await collect(node.entries())).toEqual(exp.filter((e) => e.hasOwn).map((e) => [e.key, e.own]))
expect(await collect(node.entries({ reverse: true }))).toEqual(
exp
.filter((e) => e.hasOwn)
.map((e) => [e.key, e.own])
.reverse(),
)
for (let i = 0; i < keys.length; i++) {
expect(await collect(node.keys({ after: keys[i]! }))).toEqual(keys.slice(i + 1))
expect(await collect(node.keys({ after: keys[i]!, reverse: true }))).toEqual(keys.slice(0, i).reverse())
expect(await collect(node.keys({ limit: i + 1 }))).toEqual(keys.slice(0, i + 1))
}
}
},
),
{ numRuns: 300 },
)
}, 60_000)
}
})
test/kv/sorted.test.ts
-67
View File
@@ -1,67 +0,0 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { ChunkedSortedMap } from '../../src/kv/sorted.js'
const key = fc.string({ unit: fc.constantFrom('a', 'b', 'c', '/', '0'), minLength: 1, maxLength: 4 })
type Op = { t: 'set'; k: string } | { t: 'del'; k: string } | { t: 'bulk'; ks: string[] }
const op: fc.Arbitrary<Op> = fc.oneof(
{ weight: 5, arbitrary: fc.record({ t: fc.constant('set' as const), k: key }) },
{ weight: 3, arbitrary: fc.record({ t: fc.constant('del' as const), k: key }) },
{ weight: 1, arbitrary: fc.record({ t: fc.constant('bulk' as const), ks: fc.array(key, { minLength: 60, maxLength: 120 }) }) },
)
describe('ChunkedSortedMap (model-based, tiny chunks)', () => {
it('matches a sorted reference for ranges in both directions', () => {
fc.assert(
fc.property(fc.array(op, { maxLength: 80 }), fc.integer({ min: 2, max: 6 }), key, key, (ops, chunk, a, b) => {
const m = new ChunkedSortedMap<number>(chunk)
const ref = new Map<string, number>()
ops.forEach((o, i) => {
if (o.t === 'set') (m.set(o.k, i), ref.set(o.k, i))
else if (o.t === 'del') expect(m.delete(o.k)).toBe(ref.delete(o.k))
else (m.setMany(o.ks.map((k) => [k, i])), o.ks.forEach((k) => ref.set(k, i)))
})
const sorted = [...ref.keys()].sort()
const [gte, lt] = a < b ? [a, b] : [b, a]
expect(m.size).toBe(ref.size)
expect([...m.keys({})]).toEqual(sorted)
expect([...m.keys({ reverse: true })]).toEqual([...sorted].reverse())
expect([...m.keys({ gte, lt })]).toEqual(sorted.filter((k) => k >= gte && k < lt))
expect([...m.keys({ gte, lt, reverse: true })]).toEqual(sorted.filter((k) => k >= gte && k < lt).reverse())
for (const k of sorted) expect(m.get(k)).toBe(ref.get(k))
}),
{ numRuns: 500 },
)
})
it('iterators resume correctly after mutations between steps', () => {
fc.assert(
fc.property(
fc.uniqueArray(key, { minLength: 1, maxLength: 60 }),
fc.array(fc.tuple(fc.boolean(), key), { maxLength: 40 }),
fc.boolean(),
(initial, muts, reverse) => {
const m = new ChunkedSortedMap<number>(3)
initial.forEach((k) => m.set(k, 0))
const live = new Set(initial)
const seen: string[] = []
let step = 0
for (const k of m.keys({ reverse })) {
seen.push(k)
const mut = muts[step++]
if (mut) {
const [add, mk] = mut
if (add) (m.set(mk, 1), live.add(mk))
else (m.delete(mk), live.delete(mk))
}
}
// Strictly monotone, and every key present throughout the iteration was visited.
for (let i = 1; i < seen.length; i++) expect(reverse ? seen[i]! < seen[i - 1]! : seen[i]! > seen[i - 1]!).toBe(true)
const stable = initial.filter((k) => !muts.some(([, mk]) => mk === k))
for (const k of stable) expect(seen).toContain(k)
},
),
{ numRuns: 500 },
)
})
})
test/types.test-d.ts
+39 -73
View File
@@ -1,123 +1,89 @@
import { describe, expectTypeOf, it } from 'vitest'
import { KVStorage, type KVNode } from '../src/kv/node.js'
import { InMemoryKVDriver } from '../src/kv/memory.js'
import { KVEventStorage } from '../src/events/kv.js'
import type { StoredEvent } from '../src/events/types.js'
import { EventSourcedDocumentStorage } from '../src/documents/event-sourced.js'
import type { FindResult } from '../src/documents/types.js'
import type { AnySpec, Fields, Many } from '../src/types/schema.js'
import type { BaseKVStorage } from '../src/kv-storage/base-kv-storage.js'
import { KVStorageInMemory } from '../src/kv-storage/kv-storage-in-memory.js'
import { EventStorageKV } from '../src/event-storage/event-storage-kv.js'
import type { StoredEvent } from '../src/event-storage/base-event-storage.js'
import { DocumentStorageInMemory } from '../src/document-storage/document-storage-in-memory.js'
import { EventSourcedDocumentStorage } from '../src/document-storage/event-sourced-document-storage.js'
import type { DocChange } from '../src/utils/document-update.js'
import type { FindResult } from '../src/utils/document-query.js'
import type { Fields, Many } from '../src/utils/schema.js'
type User = { username: string; age: number }
type Post = { title: string; createdAt: string }
type Schema = {
children: {
users: Many<User, { posts: Many<Post> }>
profile: { children: { [id: string]: { children: Fields<User> } } } // field-per-key style
profile: { children: { [id: string]: { children: Fields<User> } } }
}
}
const driver = new InMemoryKVDriver()
describe('KV types', () => {
const db = new KVStorageInMemory<Schema>()
describe('KV types (section 7)', () => {
const db = new KVStorage<Schema>(driver)
it('reads child values at dynamic levels', () => {
it('types reads at dynamic and named levels', () => {
expectTypeOf(db.child('users').get('123')).toEqualTypeOf<Promise<User | undefined>>()
expectTypeOf(db.child('users').child('123').get()).toEqualTypeOf<Promise<User | undefined>>()
expectTypeOf(db.child('users').getMany(['1', '2'])).toEqualTypeOf<Promise<(User | undefined)[]>>()
})
it('navigates named levels', () => {
expectTypeOf(db.child('users').child('123').child('posts')).toEqualTypeOf<KVNode<Many<Post>>>()
// @ts-expect-error unknown named child
db.child('users').child('123').child('postz')
// @ts-expect-error unknown top-level child
db.child('userz')
})
it('reads field-per-key values', () => {
expectTypeOf(db.child('profile').child('9').get('age')).toEqualTypeOf<Promise<number | undefined>>()
expectTypeOf(db.child('profile').child('9').get('username')).toEqualTypeOf<Promise<string | undefined>>()
expectTypeOf(db.child('users').child('123').child('posts')).toEqualTypeOf<BaseKVStorage<Many<Post>>>()
// @ts-expect-error unknown child
db.child('users').child('123').child('postz')
// @ts-expect-error unknown field
db.child('profile').child('9').get('agee')
})
it('checks written values', () => {
it('types writes', () => {
db.child('users').set('1', { username: 'a', age: 1 })
db.child('users').child('1').set({ username: 'a', age: 1 })
// @ts-expect-error wrong value type
// @ts-expect-error missing field
db.child('users').set('1', { username: 'a' })
db.child('profile')
.child('9')
.setMany([
['age', 3],
['username', 'x'],
])
db.child('profile')
.child('9')
.setMany([
// @ts-expect-error value doesn't match its key
['age', 'x'],
['username', 3],
])
})
it('types listings', () => {
expectTypeOf(db.child('users').keys()).toEqualTypeOf<AsyncGenerator<string>>()
expectTypeOf(db.child('profile').child('9').keys()).toEqualTypeOf<AsyncGenerator<'username' | 'age'>>()
expectTypeOf(db.child('users').entries()).toEqualTypeOf<AsyncGenerator<[string, User]>>()
})
it('defaults to untyped and supports as<S>()', () => {
const raw = new KVStorage(driver)
expectTypeOf(raw.child('anything').child(5).get('x')).toEqualTypeOf<Promise<unknown>>()
const users = raw.child('users').as<Many<User>>()
expectTypeOf(users.get('1')).toEqualTypeOf<Promise<User | undefined>>()
expectTypeOf(raw).toMatchTypeOf<KVNode<AnySpec>>()
it('is untyped by default', () => {
expectTypeOf(new KVStorageInMemory().child('a').child(1).get('b')).toEqualTypeOf<Promise<unknown>>()
})
})
type OrderEvent = { type: 'created'; total: number } | { type: 'paid' }
type EventSchema = { children: { orders: Many<OrderEvent, { shipments: Many<{ carrier: string }> }> } }
describe('event and document types', () => {
type OrderEvent = { type: 'created'; total: number } | { type: 'paid' }
const orders = new EventStorageKV<{ children: { orders: Many<OrderEvent> } }>(new KVStorageInMemory()).child('orders')
const users = new DocumentStorageInMemory<{ children: { users: Many<User, { posts: Many<Post> }> } }>().child('users')
describe('event types', () => {
const events = new KVEventStorage<EventSchema>(new KVStorage(driver))
const orders = events.child('orders')
it('checks appended events', () => {
orders.append('o1', [{ type: 'created', total: 40 }])
it('types events', () => {
orders.append('o1', [{ type: 'created', total: 1 }])
orders.child('o1').append([{ type: 'paid' }])
// @ts-expect-error not an OrderEvent
orders.append('o1', [{ type: 'refunded' }])
orders
.child('o1')
.child('shipments')
.append('s1', [{ carrier: 'x' }])
expectTypeOf(orders.read('o1')).toEqualTypeOf<AsyncGenerator<StoredEvent<OrderEvent>>>()
})
it('types read results', () => {
expectTypeOf(orders.read('o1')).toEqualTypeOf<AsyncIterable<StoredEvent<OrderEvent>>>()
expectTypeOf(orders.version('o1')).toEqualTypeOf<Promise<number>>()
})
})
type DocSchema = { children: { users: Many<User, { posts: Many<Post> }> } }
describe('document types', () => {
const docs = new EventSourcedDocumentStorage<DocSchema>(new KVEventStorage(new KVStorage(driver)))
const users = docs.child('users')
it('types reads and writes', () => {
it('types documents', () => {
expectTypeOf(users.get('1')).toEqualTypeOf<Promise<User | undefined>>()
expectTypeOf(users.child('1').get()).toEqualTypeOf<Promise<User | undefined>>()
expectTypeOf(users.insert({ username: 'a', age: 1 })).toEqualTypeOf<Promise<string>>()
// @ts-expect-error missing field
users.insert({ username: 'a' })
expectTypeOf(users.update('1', { $inc: { age: 1 } })).toEqualTypeOf<Promise<User | undefined>>()
expectTypeOf(users.child('1').child('posts').get('p')).toEqualTypeOf<Promise<Post | undefined>>()
expectTypeOf(users.find()).toEqualTypeOf<AsyncIterable<FindResult<User>>>()
// @ts-expect-error missing field
users.insert({ username: 'a' })
})
it('types queries', () => {
expectTypeOf(users.find({ age: { $gte: 18 } })).toEqualTypeOf<AsyncGenerator<FindResult<User>>>()
expectTypeOf(users.findOne()).toEqualTypeOf<Promise<FindResult<User> | undefined>>()
it('keeps event-sourced methods on children', () => {
const docs = new EventSourcedDocumentStorage<{ children: { users: Many<User> } }>(
new EventStorageKV(new KVStorageInMemory()),
new DocumentStorageInMemory(),
)
expectTypeOf(docs.child('users').history('1')).toEqualTypeOf<AsyncIterable<StoredEvent<DocChange>>>()
expectTypeOf(docs.child('users').getAt('1', 3)).toEqualTypeOf<Promise<User | undefined>>()
})
})
test/utils/utils.test.ts
+97
View File
@@ -0,0 +1,97 @@
import { describe, expect, it } from 'vitest'
import fc from 'fast-check'
import { Keys } from '../../src/utils/keys.js'
import { SortedMap } from '../../src/utils/sorted-map.js'
import { Ulid } from '../../src/utils/ulid.js'
import { DocumentQuery } from '../../src/utils/document-query.js'
import { DocumentUpdate } from '../../src/utils/document-update.js'
describe('Keys', () => {
it('round-trips any segment and never emits a character at or below the separator', () => {
fc.assert(
fc.property(fc.string({ unit: 'binary', minLength: 1 }), (s) => {
const escaped = Keys.escape(s)
expect(Keys.unescape(escaped)).toBe(s)
expect([...escaped].every((c) => c > '/')).toBe(true)
expect(Keys.isReserved(escaped)).toBe(false)
}),
)
})
it('keeps every child subtree contiguous in key order', () => {
fc.assert(
fc.property(
fc.uniqueArray(fc.string({ unit: fc.constantFrom('a', '-', '.', '/', '~', ' ', '0'), minLength: 1, maxLength: 3 }), { minLength: 2 }),
(segments) => {
const keys = segments.flatMap((s) => [Keys.escape(s), Keys.escape(s) + '/x']).sort()
const order = keys.map((k) => k.split('/')[0])
// Each segment's keys appear as one consecutive run.
expect(new Set(order).size).toBe(order.filter((s, i) => s !== order[i - 1]).length)
},
),
)
})
})
describe('SortedMap', () => {
it('matches a sorted reference, including iteration across mutations', () => {
fc.assert(
fc.property(
fc.array(fc.tuple(fc.boolean(), fc.string({ unit: fc.constantFrom('a', 'b', 'c'), maxLength: 4 })), { maxLength: 200 }),
fc.boolean(),
(ops, reverse) => {
const map = new SortedMap<number>(4)
const ref = new Set<string>()
for (const [add, key] of ops) add ? (map.set(key, 1), ref.add(key)) : (map.delete(key), ref.delete(key))
const sorted = [...ref].sort()
expect([...map.keys(undefined, undefined, reverse)]).toEqual(reverse ? sorted.reverse() : sorted)
// Deleting each key as we pass it must not skip the rest.
const seen = [...map.keys()].length
let visited = 0
for (const key of map.keys()) (visited++, map.delete(key))
expect(visited).toBe(seen)
},
),
)
})
})
describe('Ulid', () => {
it('is strictly increasing within a millisecond and sorts by time', () => {
let t = 1_700_000_000_000
const ulid = new Ulid(() => t)
const ids = Array.from({ length: 500 }, () => ulid.generate())
expect([...ids].sort()).toEqual(ids)
t++
expect(ulid.generate() > ids.at(-1)!).toBe(true)
})
})
describe('DocumentQuery', () => {
const q = (filter: object, doc: object) => new DocumentQuery(filter as never).matches(doc as never)
it('matches equality, comparisons, $in, dot paths, arrays and null-as-missing', () => {
expect(q({ a: 1 }, { a: 1 })).toBe(true)
expect(q({ 'a.b': 'x' }, { a: { b: 'x' } })).toBe(true)
expect(q({ n: { $gte: 2, $lt: 5 } }, { n: 4 })).toBe(true)
expect(q({ n: { $gt: 2 } }, { n: '9' })).toBe(false)
expect(q({ n: { $in: [1, 2] } }, { n: 2 })).toBe(true)
expect(q({ tags: 'x' }, { tags: ['y', 'x'] })).toBe(true)
expect(q({ tags: ['x'] }, { tags: ['x'] })).toBe(true)
expect(q({ a: null }, {})).toBe(true)
expect(() => q({ a: { $regex: 'x' } }, {})).toThrow(/Unsupported/)
})
})
describe('DocumentUpdate', () => {
it('resolves operators to final values', () => {
const state = { n: 5, tags: ['a', 'b'], nested: { x: 1 } }
const change = DocumentUpdate.resolve(state, { $inc: { n: 1 }, $push: { tags: { $each: ['c'] } }, $unset: { 'nested.x': 1 }, $set: { 'nested.y': 2 } })
expect(change).toEqual({ op: 'patch', set: { n: 6, tags: ['a', 'b', 'c'], 'nested.y': 2 }, unset: ['nested.x'] })
expect(DocumentUpdate.apply(state, change)).toEqual({ n: 6, tags: ['a', 'b', 'c'], nested: { y: 2 } })
expect(DocumentUpdate.apply(state, DocumentUpdate.resolve(state, { $pull: { tags: 'a' }, $addToSet: { more: { $each: [1, 1] } } }))).toMatchObject({
tags: ['b'],
more: [1],
})
expect(() => DocumentUpdate.resolve(state, { $set: { n: 1 }, $inc: { n: 1 } })).toThrow(/twice/)
})
})