Add Prettier formatting and Gitea Actions CI (format, typecheck, test, build)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
This commit is contained in:
@@ -18,7 +18,7 @@ implementation decisions made where the brief was silent (section 12) and benchm
|
||||
## 1. Principles
|
||||
|
||||
1. **Three layers**, each usable on its own: **KV** → **Event** (append-only, built on KV) → **Document** (Mongo-like, built on Events or native).
|
||||
2. **Drivers vs stores.** Backends implement a tiny, flat, string-keyed *driver*. All path, child, index, escaping and typing logic lives once in the user-facing *node* classes, so every backend gets identical behaviour.
|
||||
2. **Drivers vs stores.** Backends implement a tiny, flat, string-keyed _driver_. All path, child, index, escaping and typing logic lives once in the user-facing _node_ classes, so every backend gets identical behaviour.
|
||||
3. **One navigation primitive: `child(key)`.** There are no collection/document classes. Every node is a store over its direct children, and `child(key)` re-roots to that key. Navigation is synchronous and does no I/O.
|
||||
4. **No methods on data.** Reads return plain values.
|
||||
5. **Performance is a requirement.** Batch operations are the primitive; scans are ordered and streamed; nothing replays or scans more than it must (section 8).
|
||||
@@ -59,24 +59,22 @@ export interface ScanOptions {
|
||||
lt?: string
|
||||
limit?: number
|
||||
reverse?: boolean
|
||||
values?: boolean // when false/omitted, yield value as undefined
|
||||
values?: boolean // when false/omitted, yield value as undefined
|
||||
}
|
||||
|
||||
export interface KVDriver {
|
||||
getMany(keys: string[]): Promise<(unknown | undefined)[]> // same order as keys
|
||||
getMany(keys: string[]): Promise<(unknown | undefined)[]> // same order as keys
|
||||
setMany(entries: [key: string, value: unknown][]): Promise<void>
|
||||
deleteMany(keys: string[]): Promise<void>
|
||||
scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]> // MUST be key-ordered
|
||||
scan(opts: ScanOptions): AsyncIterable<[key: string, value: unknown]> // MUST be key-ordered
|
||||
|
||||
// Optional capabilities. Stores feature-detect them and fall back where possible.
|
||||
setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]> // true = written
|
||||
batch?(ops: BatchOp[]): Promise<void> // atomic multi-op write
|
||||
setIfAbsent?(entries: [string, unknown][]): Promise<boolean[]> // true = written
|
||||
batch?(ops: BatchOp[]): Promise<void> // atomic multi-op write
|
||||
deleteRange?(gte: string, lt: string): Promise<void>
|
||||
}
|
||||
|
||||
export type BatchOp =
|
||||
| { type: 'set'; key: string; value: unknown }
|
||||
| { type: 'delete'; key: string }
|
||||
export type BatchOp = { type: 'set'; key: string; value: unknown } | { type: 'delete'; key: string }
|
||||
```
|
||||
|
||||
- Values are passed through as JSON-serialisable values. Drivers that need bytes (HTTP, Redis, files) serialise internally via a pluggable `Codec` (default JSON).
|
||||
@@ -94,29 +92,31 @@ const user = users.child('123')
|
||||
|
||||
API of every `KVNode` (types in section 7):
|
||||
|
||||
| Method | Meaning |
|
||||
|---|---|
|
||||
| `child(key)` | Sync. Returns a node at `path + key`. |
|
||||
| `as<S>()` | Type assertion for untyped use; returns the same node retyped. |
|
||||
| `get()` / `get(key)` | Own value at `selfKey` / value at child key. |
|
||||
| `getMany(keys)` | Batch child read, same order, `undefined` for missing. |
|
||||
| `set(value)` / `set(key, value)` | Write own value / child value. |
|
||||
| `setMany(entries)` | Batch child write. |
|
||||
| `delete()` / `delete(key)` | Delete own value / child value only (not descendants). |
|
||||
| `deleteMany(keys)` | Batch child delete. |
|
||||
| `deleteTree()` / `deleteTree(key)` | Delete own value **and all descendants** (uses `deleteRange` if available). |
|
||||
| `keys(opts?)` | `AsyncIterable<string>` of **direct** child keys (unescaped). Options: `limit`, `after`, `reverse`. |
|
||||
| `entries(opts?)` | `AsyncIterable<[key, value]>` of direct children that have an own value. |
|
||||
| `tree({ depth })` | Assemble an object from descendant keys up to `depth` (for field-per-key style). Bounded and explicit; never the default. |
|
||||
| `findBy(index, value \| { gte?, lt? }, opts?)` | Index lookup over direct children (section 3.3). |
|
||||
| Method | Meaning |
|
||||
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `child(key)` | Sync. Returns a node at `path + key`. |
|
||||
| `as<S>()` | Type assertion for untyped use; returns the same node retyped. |
|
||||
| `get()` / `get(key)` | Own value at `selfKey` / value at child key. |
|
||||
| `getMany(keys)` | Batch child read, same order, `undefined` for missing. |
|
||||
| `set(value)` / `set(key, value)` | Write own value / child value. |
|
||||
| `setMany(entries)` | Batch child write. |
|
||||
| `delete()` / `delete(key)` | Delete own value / child value only (not descendants). |
|
||||
| `deleteMany(keys)` | Batch child delete. |
|
||||
| `deleteTree()` / `deleteTree(key)` | Delete own value **and all descendants** (uses `deleteRange` if available). |
|
||||
| `keys(opts?)` | `AsyncIterable<string>` of **direct** child keys (unescaped). Options: `limit`, `after`, `reverse`. |
|
||||
| `entries(opts?)` | `AsyncIterable<[key, value]>` of direct children that have an own value. |
|
||||
| `tree({ depth })` | Assemble an object from descendant keys up to `depth` (for field-per-key style). Bounded and explicit; never the default. |
|
||||
| `findBy(index, value \| { gte?, lt? }, opts?)` | Index lookup over direct children (section 3.3). |
|
||||
|
||||
**Arity dispatch (important):**
|
||||
|
||||
- Self vs child is decided by `arguments.length`, never by checking for `undefined`.
|
||||
- If a key argument is passed but is `undefined` or `null`, throw a `TypeError` (this prevents `users.get(maybeId)` silently reading self).
|
||||
- The root node (`path = []`) has no own value: `get()`, `set(v)` and `delete()` on root throw.
|
||||
- Implement one private `#resolve(args)` that every method uses.
|
||||
|
||||
**Direct-children listing (`keys`)** uses a skip-scan:
|
||||
|
||||
1. Scan `gte: prefix`, `lt: prefix + succ(sep)`.
|
||||
2. For each key, take the first segment after `prefix`. Skip reserved (`~…`) segments. Yield the segment (unescaped) if it differs from the previous one.
|
||||
3. When the next key is a descendant of the segment just yielded, re-seek the driver to `gte: prefix + escSeg + succ(sep)` (where `succ(c)` is the next char code) instead of iterating through the subtree. Keep using the same iterator while keys are siblings.
|
||||
@@ -130,8 +130,8 @@ Indexes are declared on **path patterns** in the `KVStorage` constructor. A patt
|
||||
```ts
|
||||
new KVStorage(driver, {
|
||||
indexes: {
|
||||
'users': { byEmail: (u: User) => u.email },
|
||||
'users/*/posts': { byDate: (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
|
||||
users: { byEmail: (u: User) => u.email },
|
||||
'users/*/posts': { byDate: (p: Post) => p.createdAt, byTag: (p: Post) => p.tags },
|
||||
},
|
||||
})
|
||||
```
|
||||
@@ -155,14 +155,17 @@ new KVStorage(driver, {
|
||||
An append-only log per stream. A stream is identified by a node path. Same `child()` navigation as KV.
|
||||
|
||||
```ts
|
||||
const events = new EventStorage(kv) // takes a KVStorage (or a KVNode as root)
|
||||
const events = new EventStorage(kv) // takes a KVStorage (or a KVNode as root)
|
||||
const orders = events.child('orders')
|
||||
|
||||
await orders.append('o1', [{ type: 'created', total: 40 }], { expectedVersion: 0 })
|
||||
await orders.child('o1').append([{ type: 'paid' }]) // self form
|
||||
for await (const e of orders.read('o1', { from: 1 })) { /* { version, commit, at, data } */ }
|
||||
await orders.child('o1').append([{ type: 'paid' }]) // self form
|
||||
for await (const e of orders.read('o1', { from: 1 })) {
|
||||
/* { version, commit, at, data } */
|
||||
}
|
||||
await orders.version('o1')
|
||||
for await (const id of orders.keys()) { } // child streams
|
||||
for await (const id of orders.keys()) {
|
||||
} // child streams
|
||||
orders.child('o1').child('shipments')
|
||||
```
|
||||
|
||||
@@ -186,23 +189,25 @@ orders.child('o1').child('shipments')
|
||||
## 5. Layer 3: Documents (`src/documents/`)
|
||||
|
||||
`DocumentStorage` is the interface. There are two implementations:
|
||||
|
||||
- `EventSourcedDocumentStorage(events, { projections?: KVNode })` is the v1 target. Projections default to the `~p` root on the same KV.
|
||||
- `MongoDocumentStorage(client)` is a later milestone (section 9).
|
||||
|
||||
```ts
|
||||
const docs = new EventSourcedDocumentStorage(events, { indexes: { 'users': [{ fields: ['email'], unique: true }, { fields: ['age'] }] } })
|
||||
const docs = new EventSourcedDocumentStorage(events, { indexes: { users: [{ fields: ['email'], unique: true }, { fields: ['age'] }] } })
|
||||
const users = docs.child('users')
|
||||
|
||||
const id = await users.insert({ name: 'Ada', age: 36 }) // generates a time-sortable id
|
||||
const id = await users.insert({ name: 'Ada', age: 36 }) // generates a time-sortable id
|
||||
await users.set('123', { name: 'Ada', age: 36 })
|
||||
await users.update('123', { $set: { name: 'Ada L.' }, $inc: { logins: 1 } })
|
||||
await users.child('123').update({ $unset: { nickname: true } }) // self form
|
||||
await users.child('123').update({ $unset: { nickname: true } }) // self form
|
||||
await users.get('123')
|
||||
await users.delete('123')
|
||||
for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) { }
|
||||
for await (const { key, data } of users.find({ age: { $gte: 18 } }, { sort: { age: 1 }, limit: 20 })) {
|
||||
}
|
||||
await users.findOne({ email: 'ada@x.io' })
|
||||
await users.history('123') // event-sourced only
|
||||
await users.getAt('123', 17) // state as of version 17
|
||||
await users.history('123') // event-sourced only
|
||||
await users.getAt('123', 17) // state as of version 17
|
||||
users.child('123').child('posts') // nested docs, same as KV
|
||||
```
|
||||
|
||||
@@ -214,6 +219,7 @@ users.child('123').child('posts') // nested docs, same as KV
|
||||
- Supported update operators in v1: `$set`, `$unset`, `$inc`, `$push`, `$pull`, `$addToSet`, with dot-path support.
|
||||
|
||||
**Write path:**
|
||||
|
||||
1. Read the projection.
|
||||
2. Compute the new state and patch.
|
||||
3. `append` with `expectedVersion = projection.version`.
|
||||
@@ -223,6 +229,7 @@ users.child('123').child('posts') // nested docs, same as KV
|
||||
**Reads** are a single projection `get`. Never replay on read.
|
||||
|
||||
**Crash between append and projection write:**
|
||||
|
||||
- The projection lags the stream. The next write's `expectedVersion` fails, which triggers catch-up, so the system self-heals.
|
||||
- Provide `get(key, { consistent: true })` that checks the stream head and catches up first.
|
||||
- Provide a `catchUp(node?)` maintenance method to repair a subtree.
|
||||
@@ -257,16 +264,15 @@ Each node is typed by a single **Spec** that describes its own value and its chi
|
||||
|
||||
```ts
|
||||
export type Spec = { value?: unknown; children?: Record<string, Spec> }
|
||||
export type AnySpec = { value: unknown; children: Record<string, AnySpec> } // untyped default
|
||||
export type AnySpec = { value: unknown; children: Record<string, AnySpec> } // untyped default
|
||||
|
||||
export type ValueOf<S> = S extends { value: infer V } ? V : never
|
||||
export type ValueOf<S> = S extends { value: infer V } ? V : never
|
||||
export type ChildrenOf<S> = S extends { children: infer C } ? C : {}
|
||||
export type KeyOf<S> = Extract<keyof ChildrenOf<S>, string | number>
|
||||
export type KeyOf<S> = Extract<keyof ChildrenOf<S>, string | number>
|
||||
export type ChildOf<S, K extends KeyOf<S>> = ChildrenOf<S>[K]
|
||||
|
||||
// Helpers
|
||||
export type Many<V, C extends Record<string, Spec> = {}> =
|
||||
{ children: { [id: string]: { value: V; children: C } } }
|
||||
export type Many<V, C extends Record<string, Spec> = {}> = { children: { [id: string]: { value: V; children: C } } }
|
||||
export type Fields<T> = { [K in keyof T & string]: { value: T[K] } }
|
||||
```
|
||||
|
||||
@@ -286,21 +292,24 @@ Example to cover with `expectTypeOf` tests:
|
||||
```ts
|
||||
type User = { username: string; age: number }
|
||||
type Post = { title: string; createdAt: string }
|
||||
type Schema = { children: {
|
||||
users: Many<User, { posts: Many<Post> }>
|
||||
profile: { children: { [id: string]: { children: Fields<User> } } } // field-per-key style
|
||||
} }
|
||||
type Schema = {
|
||||
children: {
|
||||
users: Many<User, { posts: Many<Post> }>
|
||||
profile: { children: { [id: string]: { children: Fields<User> } } } // field-per-key style
|
||||
}
|
||||
}
|
||||
|
||||
const db = new KVStorage<Schema>(driver)
|
||||
db.child('users').get('123') // User | undefined
|
||||
db.child('users').child('123').get() // User | undefined (same key)
|
||||
db.child('users').child('123').child('posts') // KVNode<Many<Post>>
|
||||
db.child('users').child('123').child('postz') // ✗ type error
|
||||
db.child('profile').child('9').get('age') // number | undefined
|
||||
db.child('profile').child('9').get('agee') // ✗ type error
|
||||
db.child('users').get('123') // User | undefined
|
||||
db.child('users').child('123').get() // User | undefined (same key)
|
||||
db.child('users').child('123').child('posts') // KVNode<Many<Post>>
|
||||
db.child('users').child('123').child('postz') // ✗ type error
|
||||
db.child('profile').child('9').get('age') // number | undefined
|
||||
db.child('profile').child('9').get('agee') // ✗ type error
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- **Named levels** check keys at compile time; **dynamic `[id: string]` levels** accept any string.
|
||||
- One node's children can't mix an index signature with named keys (a TS restriction), so schemas alternate between dynamic and named levels. That alternation exists only in types; the runtime API is just `child()`.
|
||||
- `EventStorage` and `DocumentStorage` nodes use the same Spec machinery (the value type is the event type or the document type respectively).
|
||||
@@ -383,7 +392,7 @@ Decisions taken where the brief was silent. Each is the simplest option consiste
|
||||
### M2 — KV driver
|
||||
|
||||
- **Conformance suite location**: `src/testing/kv-conformance.ts`, exported as `potential-storage/testing`.
|
||||
It imports `vitest`, which is an *optional peer dependency*; the core entry point never imports it.
|
||||
It imports `vitest`, which is an _optional peer dependency_; the core entry point never imports it.
|
||||
- **`InMemoryKVDriver` clones values** on write and read (`structuredClone`, skipped for primitives) so
|
||||
stored state can't be mutated through shared references, matching serialising drivers. Opt out
|
||||
with `{ clone: false }`.
|
||||
@@ -402,7 +411,7 @@ Decisions taken where the brief was silent. Each is the simplest option consiste
|
||||
- **Child order is "subtree order", not plain segment order.** The brief's skip-scan assumes a
|
||||
child's keys are contiguous, but they aren't when a sibling extends a segment with a character
|
||||
that sorts below the separator (for `/`: space and `!"#$%&'()*+,-.`). With children `a` and `a-1`
|
||||
the key space is `p/a`, `p/a-1`, `p/a-1/x`, `p/a/x`, so `a`'s descendants sort *after* `a-1`, and
|
||||
the key space is `p/a`, `p/a-1`, `p/a-1/x`, `p/a/x`, so `a`'s descendants sort _after_ `a-1`, and
|
||||
"yield if different from the previous segment" would emit `a` twice. `keys()`/`entries()` therefore
|
||||
order children by **escaped segment followed by the separator** (`a-1/` < `a/`), which is the order
|
||||
of their subtrees in key space. For keys without such characters this is plain lexicographic order.
|
||||
@@ -541,7 +550,7 @@ Decisions taken where the brief was silent. Each is the simplest option consiste
|
||||
field, both for index-ordered and in-memory sorts.
|
||||
- **In-memory sort** keeps a bounded top `skip + limit` buffer when a limit is given. Without a limit,
|
||||
it buffers all matches; this is the one place results are materialised.
|
||||
- **Full-scan warning**: `console.warn` once per filter *shape* (values replaced by `?`) per storage
|
||||
- **Full-scan warning**: `console.warn` once per filter _shape_ (values replaced by `?`) per storage
|
||||
instance, only for non-empty filters. On by default unless `NODE_ENV === 'production'`; set it with
|
||||
`warnOnFullScan`.
|
||||
- **Catch-up** re-asserts unique claims for the caught-up state with plain writes, since the log is
|
||||
@@ -609,37 +618,37 @@ container (shared CPU, so expect ±10–20% run-to-run). Mean time per operation
|
||||
|
||||
**KV** (`bench/kv.bench.ts`): values are small objects.
|
||||
|
||||
| Operation | 10k keys | 100k keys |
|
||||
|---|---|---|
|
||||
| `get` | 1.2 µs | 1.6 µs |
|
||||
| `set` (overwrite) | 1.7 µs | 2.0 µs |
|
||||
| `getMany` × 100 keys | 21 µs | 22 µs |
|
||||
| Operation | 10k keys | 100k keys |
|
||||
| -------------------- | -------- | --------- |
|
||||
| `get` | 1.2 µs | 1.6 µs |
|
||||
| `set` (overwrite) | 1.7 µs | 2.0 µs |
|
||||
| `getMany` × 100 keys | 21 µs | 22 µs |
|
||||
|
||||
**`keys()`** (`bench/keys.bench.ts`): node with 1 000 children, each with 100 descendants (101k keys).
|
||||
|
||||
| Operation | Mean |
|
||||
|---|---|
|
||||
| Operation | Mean |
|
||||
| -------------------------------- | ------ |
|
||||
| `keys()` skip-scan (1 000 seeks) | 2.9 ms |
|
||||
| Reference: full subtree scan | 36 ms |
|
||||
| Reference: full subtree scan | 36 ms |
|
||||
|
||||
**Events** (`bench/events.bench.ts`): single-event commits.
|
||||
|
||||
| Operation | Mean |
|
||||
|---|---|
|
||||
| `append` 1 event (warm head cache) | 6.3 µs |
|
||||
| Operation | Mean |
|
||||
| ---------------------------------------------- | ------ |
|
||||
| `append` 1 event (warm head cache) | 6.3 µs |
|
||||
| `read` 100 events from version 5 000 of 10 000 | 174 µs |
|
||||
| `read` full 10 000-event stream | 21 ms |
|
||||
| `version()` (reverse scan, `limit: 1`) | 3.9 µs |
|
||||
| `read` full 10 000-event stream | 21 ms |
|
||||
| `version()` (reverse scan, `limit: 1`) | 3.9 µs |
|
||||
|
||||
**Documents** (`bench/documents.bench.ts`): 10 000 docs, indexes on `age` and unique `email`.
|
||||
|
||||
| Operation | Mean |
|
||||
|---|---|
|
||||
| `get` | 2.6 µs |
|
||||
| `update` (`$inc`, indexed node: projection read + append + batch) | 22 µs |
|
||||
| `findOne` by unique index (`email`) | 16 µs |
|
||||
| `find` indexed (`age = k`, ~100 hits) | 0.57 ms |
|
||||
| `find` full scan (`city = k`, ~200 hits of 10k) | 28 ms |
|
||||
| Operation | Mean |
|
||||
| ----------------------------------------------------------------- | ------- |
|
||||
| `get` | 2.6 µs |
|
||||
| `update` (`$inc`, indexed node: projection read + append + batch) | 22 µs |
|
||||
| `findOne` by unique index (`email`) | 16 µs |
|
||||
| `find` indexed (`age = k`, ~100 hits) | 0.57 ms |
|
||||
| `find` full scan (`city = k`, ~200 hits of 10k) | 28 ms |
|
||||
|
||||
For comparison, the first run with the sorted-array map and `structuredClone` measured: KV `get`
|
||||
3.7 µs, `getMany`×100 186 µs, `append` 34 µs, document `update` 110 µs, full scan 57 ms, and
|
||||
|
||||
Reference in New Issue
Block a user