diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 9658d3d..b7c2b70 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -32,11 +32,26 @@ jobs: - name: Build run: npm run build + browser: + name: Chromium + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: npm + - run: npm ci + - name: Install Chromium + run: npx playwright install --with-deps chromium + - name: Browser tests + run: npm run test:browser + # Pushes only (not pull requests): main → `latest`, next → `next`, other branches → `dev-`. # See scripts/publish.mjs. Needs the NPM_REGISTRY_TOKEN secret. staging: name: Publish to staging registry - needs: check + needs: [check, browser] if: github.event_name == 'push' runs-on: ubuntu-latest steps: diff --git a/CLAUDE.md b/CLAUDE.md index 507cc91..4ec6aca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,10 +7,14 @@ Layered TypeScript storage library: KV → Events → Documents. The design is i - `npm test`: `tsc --noEmit`, then all Vitest tests (including `test/types.test-d.ts`) - `npm run test:only -- `: Vitest only +- `npm run test:browser`: `test/browser/` in headless Chromium (Vitest browser mode + Playwright) + against a real server on another origin (`test/browser/global-setup.ts`) - `npm run typecheck`, `npm run build`, `npm run bench` - `npm run format` / `npm run format:check`: Prettier -CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs the format check, typecheck, tests and build on Node 22, 24 and 26. +CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs the format check, typecheck, tests and build on +Node 22, 24 and 26, and the Chromium tests. On pushes it then publishes to the staging registry +(`scripts/publish.mjs`: `main` → `latest`, `next` → `next`, others → `dev-`). ## Conventions @@ -23,6 +27,7 @@ CI (`.gitea/workflows/ci.yml`, Gitea Actions) runs the format check, typecheck, object or in Maps (see `KVStorageComposite`, `KVStorageHttp`). - No runtime dependencies; third-party code is vendored (`src/utils/sse-session/`, `event-emitter.ts`, `extended-json.ts`, with a header naming the source). Tests live only in `test/`, mirroring `src/`. +- `README.md` examples are mirrored in `test/readme.test.ts`; change both together. - New KV/document implementations must pass `test/kv-storage/kv-storage.conformance.ts` / `test/document-storage/document-storage.suite.ts`. - Ask the owner before pushing. diff --git a/DESIGN.md b/DESIGN.md index 64fe994..dae767e 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -33,13 +33,16 @@ section 7 lists what changed and why. ``` src/ kv-storage/ base-kv-storage.ts, kv-storage-in-memory.ts, kv-storage-indexeddb.ts, - kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts + kv-storage-http.ts, kv-storage-http-server.ts, kv-storage-composite.ts, + kv-storage-encrypted.ts event-storage/ base-event-storage.ts, event-storage-kv.ts document-storage/ base-document-storage.ts, document-storage-in-memory.ts, document-storage-kv.ts, event-sourced-document-storage.ts utils/ keys, sorted-map, json, extended-json, event-emitter, ulid, errors, schema (types), document-path/query/update, sse-session/ (vendored) -test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts +test/ mirrors src/; shared suites: kv-storage.conformance.ts, document-storage.suite.ts; + browser/ runs in headless Chromium against a real cross-origin server; + readme.test.ts runs the README's examples bench/ ``` @@ -99,7 +102,10 @@ no own value. A node's own value and its children are independent keys. `scan` streams NDJSON. Unreachable server (fetch failure, 502/503/504) → `StorageUnavailableError`. The server executes exactly what clients send and never reads values for decisions, so values can be encrypted end to end. - - `KVStorageComposite`: see 4.2. + - `KVStorageComposite`: see 4.2. `KVStorageEncrypted`: see 4.3. +- **CORS.** `allowOrigins` (`['https://app…']` or `'*'`; default none) answers preflights and adds + `Access-Control-Allow-Origin` for allowed origins. Allowed request headers: `content-type`, + `cache-control` (sent by `SSESession`), `last-event-id` and the auth headers. - **Change stream (SSE).** `POST /events` with `{ paths }` streams every change to those keys and their subtrees as `event: change` (data: the changes), including the subscriber's own writes. `event: ready` follows each connection; keep-alive comments are sent every 15 s. The server keeps @@ -109,6 +115,8 @@ no own value. A node's own value and its children are independent keys. new list; `subscribe` resolves once connected. A 401/403 rejects instead of retrying. - The client emits each pushed batch as `change` (`origin: 'remote'`), and `resync` whenever the stream reconnects after a drop, since changes may have been missed meanwhile. + - One SSE connection per `KVStorageHttp` instance. Browsers allow 6 HTTP/1.1 connections per + host, so apps should share one client per server (or use HTTP/2); `close()` releases it. - Limitation: a server only sees writes made through its own storage instance, so several server processes over one database need a shared pub/sub (not built). - **Wrappers** (e.g. a future encryption layer) extend `BaseKVStorage` and delegate the primitives. @@ -159,20 +167,41 @@ is only appropriate when the client may read all of it. - A write the source refuses (e.g. 403) is undone locally and rethrown. - On `StorageUnavailableError`, or while a sync runs or writes are queued (to keep order), the write goes to the queue and is sent later. The last write to reach the source wins. + - A queued write the source refuses on replay is dropped (retrying would block every later + write), the source's values for its keys are restored locally, and it is reported to `onError`. - **Remote changes** pushed by the source are applied and announced (`change`, `origin: 'remote'`) only where they differ from what the first replica holds. No write tagging or echo tracking is needed: echoes of this client's writes change nothing. - **`sync()`**: replay the queue in order (send, then re-apply locally) → merge-scan the source against the last replica (both key-ordered), applying the difference in batches of 500 → replay again (writes made meanwhile) → mark `synced`. It runs on start, on `resync` (the stream - reconnected), and `retryMs` after a failure while offline. + reconnected), and `retryMs` after a failure while offline. Called while a sync is running, it + schedules one more after it (shared by all callers meanwhile), so changes after the running + sync's scan aren't missed; e.g. the sync after subscribing, which closes the gap between the + first scan and the stream connecting. - **Reads** wait for the first sync. Until one has completed, they go to the source. With a persistent queue, a restart reads locally straight away (offline too). -- `start()` (also lazy on first use), `sync()`, `queued`, `close()`. +- `start()` (also lazy on first use), `sync()`, `queued`, `close()`; `onError` receives errors no + caller can be rejected with (refused queued writes, failed syncs). Default `console.error`. - Limitations: catching up re-scans the whole range (no server history); a remote change can overwrite a value whose write is still queued until that write is replayed (last write wins); replicas, source and queue must be root stores (the primitives use absolute keys). +### 4.3 `KVStorageEncrypted(inner, { key })` + +Encrypts values before they reach another KV storage (usually `KVStorageHttp`), so the server only +stores ciphertext. + +- AES-GCM (WebCrypto) with a fresh 96-bit random IV per write. Stored value: `Uint8Array` + `[1 (format), iv (12), ciphertext + tag]`. The storage key is the additional authenticated data, + so a value moved to another key, or altered, fails to decrypt (`Error`, with the cause). +- Keys stay plaintext (the server needs them for authorization, ordering and prefix scans); value + sizes and timing are visible. +- Remote `change` events from the inner storage are decrypted (in order) and re-emitted; `resync` + and `subscribe` / `unsubscribe` pass through, so it works as a composite `source`. +- Keys: `generateKey()` (extractable), `exportKey` / `importKey` (raw), `deriveKey(password, salt, +iterations = 600 000)` (PBKDF2-SHA-256, not extractable). No key rotation yet. + ## 5. Event storage `BaseEventStorage` primitives over stream paths: `appendTo(stream, events)` → ids, @@ -258,6 +287,8 @@ Node API: `get(key?)`, `getMany`, `set(doc)` / `set(key, doc)`, `insert(doc)` re-scan (would need server-side history or a change log in event storage). - Event-sourced documents: subscribing to event streams to run `catchUp` automatically. - Authorization beyond "public key = user id" (sharing a subtree with other keys). +- Encryption: key rotation (re-encrypt, or a key id in the value format); sharing encrypted data + between users. - Multi-process servers need a shared change feed (pub/sub) for SSE. - Hard deletes from immutable event logs (privacy requests): compaction vs crypto-shredding. The diff --git a/README.md b/README.md new file mode 100644 index 0000000..a98ee43 --- /dev/null +++ b/README.md @@ -0,0 +1,293 @@ +# @harvmaster/storage + +Layered TypeScript storage that works the same in the browser and on the server: + +- **KV storage**: ordered keys and values, navigated as a tree (`db.child('users').get('1')`). + In memory, IndexedDB, or over HTTP, with live updates over Server-Sent Events. +- **Event storage**: append-only streams on top of any KV storage. +- **Document storage**: JSON documents with queries, updates (`$set`, `$inc`, …) and indexes, on + top of any KV storage, or event-sourced. +- **Sync**: local replicas that work offline and stay up to date with a server + (`KVStorageComposite`), end-to-end encryption (`KVStorageEncrypted`) and signed requests where + the user's public key is their id. + +No runtime dependencies. Node 22+ and current browsers. Values are JSON plus `bigint` and +`Uint8Array`. + +> **Status: experimental (0.x).** The API may change between minor versions. See +> [DESIGN.md](./DESIGN.md) for how everything works and its known limitations. + +## Install + +The package is published to a private registry. Add to your project's `.npmrc`: + +```ini +@harvmaster:registry=https://verdaccio.harvmaster.com/ +``` + +```sh +npm install @harvmaster/storage # latest release +npm install @harvmaster/storage@next # `next` branch +npm install @harvmaster/storage@dev-sync # a development branch (`dev-`) +``` + +## KV storage + +Every store is a tree of nodes. `child(key)` returns a view rooted deeper (no I/O); methods take +an optional key to act on a child instead of the node itself. + +```ts +import { KVStorageInMemory } from '@harvmaster/storage' + +const db = new KVStorageInMemory() +const users = db.child('users') + +await users.set('1', { name: 'Ada' }) +await users.child('1').get() // { name: 'Ada' } +await users.get('1') // the same +await users.setMany([ + ['2', { name: 'Bob' }], + ['3', { name: 'Cy' }], +]) +await users.getMany(['1', 'missing']) // [{ name: 'Ada' }, undefined] + +for await (const key of users.keys({ limit: 10 })) console.log(key) // '1', '2', '3' +for await (const [key, user] of users.entries({ reverse: true })) console.log(key, user) + +await users.delete('3') +await users.deleteTree() // users and everything under it +``` + +A node's own value and its children are independent, so an object can also be stored field per key: + +```ts +const settings = db.child('settings') +await settings.setFields({ theme: 'dark', fontSize: 14 }) +await settings.get('theme') // 'dark' +await settings.getFields() // { fontSize: 14, theme: 'dark' } +``` + +Keys sort as strings (`'10'` before `'9'`), and any string or number is a valid key: special +characters are escaped. Every write emits a `change` event: + +```ts +db.on('change', ({ changes, origin }) => console.log(origin, changes)) // 'local', [{ key: 'users/1', value: … }] +``` + +### Types + +Describe the tree once to get typed keys and values: + +```ts +import { KVStorageInMemory, type Many, type Fields } from '@harvmaster/storage' + +type User = { name: string } +type Schema = { children: { users: Many; settings: { children: Fields<{ theme: string; fontSize: number }> } } } + +const typed = new KVStorageInMemory() +await typed.child('users').set('1', { name: 'Ada' }) // checked +const theme = await typed.child('settings').get('theme') // string | undefined +``` + +### IndexedDB + +```ts +import { KVStorageIndexedDB } from '@harvmaster/storage' + +const local = new KVStorageIndexedDB({ name: 'my-app' }) +``` + +## Event storage + +Append-only streams. Every node is a stream (its own events) and a parent of other streams. +Appends never conflict: concurrent writers each add their own events, and every reader sees all of +them in the same (time) order. + +```ts +import { EventStorageKV, KVStorageInMemory } from '@harvmaster/storage' + +const events = new EventStorageKV(new KVStorageInMemory()) +const orders = events.child('orders') + +const [id] = await orders.append('o1', [{ type: 'created', total: 30 }]) +await orders.child('o1').append([{ type: 'paid' }]) + +for await (const event of orders.read('o1')) console.log(event.id, event.at, event.data) +for await (const event of orders.read('o1', { after: id })) console.log(event.data) // { type: 'paid' } +for await (const key of orders.keys()) console.log(key) // 'o1' +``` + +Event ids are `