PotentialStorage/README.md

# @harvmaster/storage

Layered TypeScript storage that works the same in the browser and on the server:

- **KV storage**: ordered keys and values, navigated as a tree (`db.child('users').get('1')`).
  In memory, IndexedDB, or over HTTP, with live updates over Server-Sent Events.
- **Event storage**: append-only streams on top of any KV storage.
- **Document storage**: JSON documents with queries, updates (`$set`, `$inc`, …) and indexes, on
  top of any KV storage, or event-sourced.
- **Sync**: local replicas that work offline and stay up to date with a server
  (`KVStorageComposite`), end-to-end encryption (`KVStorageEncrypted`) and signed requests where
  the user's public key is their id.

No runtime dependencies. Node 22+ and current browsers. Values are JSON plus `bigint` and
`Uint8Array`.

> **Status: experimental (0.x).** The API may change between minor versions. See
> [DESIGN.md](./DESIGN.md) for how everything works and its known limitations.

## Install

The package is published to a private registry. Add to your project's `.npmrc`:

```ini
@harvmaster:registry=https://verdaccio.harvmaster.com/
```

```sh
npm install @harvmaster/storage          # latest release
npm install @harvmaster/storage@next     # `next` branch
npm install @harvmaster/storage@dev-sync # a development branch (`dev-<branch>`)
```

## KV storage

Every store is a tree of nodes. `child(key)` returns a view rooted deeper (no I/O); methods take
an optional key to act on a child instead of the node itself.

```ts
import { KVStorageInMemory } from '@harvmaster/storage'

const db = new KVStorageInMemory()
const users = db.child('users')

await users.set('1', { name: 'Ada' })
await users.child('1').get() // { name: 'Ada' }
await users.get('1') // the same
await users.setMany([
  ['2', { name: 'Bob' }],
  ['3', { name: 'Cy' }],
])
await users.getMany(['1', 'missing']) // [{ name: 'Ada' }, undefined]

for await (const key of users.keys({ limit: 10 })) console.log(key) // '1', '2', '3'
for await (const [key, user] of users.entries({ reverse: true })) console.log(key, user)

await users.delete('3')
await users.deleteTree() // users and everything under it
```

A node's own value and its children are independent, so an object can also be stored field per key:

```ts
const settings = db.child('settings')
await settings.setFields({ theme: 'dark', fontSize: 14 })
await settings.get('theme') // 'dark'
await settings.getFields() // { fontSize: 14, theme: 'dark' }
```

Keys sort as strings (`'10'` before `'9'`), and any string or number is a valid key: special
characters are escaped. Every write emits a `change` event:

```ts
db.on('change', ({ changes, origin }) => console.log(origin, changes)) // 'local', [{ key: 'users/1', value: … }]
```

### Types

Describe the tree once to get typed keys and values:

```ts
import { KVStorageInMemory, type Many, type Fields } from '@harvmaster/storage'

type User = { name: string }
type Schema = { children: { users: Many<User>; settings: { children: Fields<{ theme: string; fontSize: number }> } } }

const typed = new KVStorageInMemory<Schema>()
await typed.child('users').set('1', { name: 'Ada' }) // checked
const theme = await typed.child('settings').get('theme') // string | undefined
```

### IndexedDB

```ts
import { KVStorageIndexedDB } from '@harvmaster/storage'

const local = new KVStorageIndexedDB({ name: 'my-app' })
```

## Event storage

Append-only streams. Every node is a stream (its own events) and a parent of other streams.
Appends never conflict: concurrent writers each add their own events, and every reader sees all of
them in the same (time) order.

```ts
import { EventStorageKV, KVStorageInMemory } from '@harvmaster/storage'

const events = new EventStorageKV(new KVStorageInMemory())
const orders = events.child('orders')

const [id] = await orders.append('o1', [{ type: 'created', total: 30 }])
await orders.child('o1').append([{ type: 'paid' }])

for await (const event of orders.read('o1')) console.log(event.id, event.at, event.data)
for await (const event of orders.read('o1', { after: id })) console.log(event.data) // { type: 'paid' }
for await (const key of orders.keys()) console.log(key) // 'o1'
```

Event ids are `<time>_<hash of the time and content>`, so writing the same event again (e.g. a
retried request) can't duplicate it.

## Document storage

```ts
import { DocumentStorageKV, KVStorageInMemory } from '@harvmaster/storage'

const docs = new DocumentStorageKV(new KVStorageInMemory())
const people = docs.child('people')

const id = await people.insert({ name: 'Ada', age: 36, tags: ['math'] })
await people.set('grace', { name: 'Grace', age: 45, tags: ['navy', 'cobol'] })
await people.update(id, { $inc: { age: 1 }, $push: { tags: 'engines' } })

for await (const { key, data } of people.find({ age: { $gte: 37 } }, { sort: { age: -1 }, limit: 10 })) console.log(key, data)
await people.findOne({ tags: 'cobol' }) // { key: 'grace', data: { … } }
await people.delete('grace') // true
```

Filters: `{ field: value }` (dot paths; arrays match any element), `$eq $gt $gte $lt $lte $in`.
Updates: `$set $unset $inc $push $pull $addToSet` (`$each` for push/addToSet).

- `DocumentStorageKV` stores each document as one KV value, so it works over any KV storage
  (including a synced one, below). Queries scan the collection.
- `DocumentStorageInMemory({ indexes: { people: ['age'] } })` keeps equality indexes.
- `EventSourcedDocumentStorage(events, projections)` records every change as an event (with
  `history(key)` and `getAt(key, eventId)`), and keeps the current state in any document store.

## Sync, offline and encryption

The pieces below combine into an offline-first app whose server only stores ciphertext.

### Server

`KVStorageHttpServer` serves any KV storage over HTTP. `handle(request)` takes a Fetch `Request`
and returns a `Response`, so it works with `Deno.serve`, `Bun.serve`, Cloudflare Workers, Hono,
and so on (in Node, through an adapter such as `@hono/node-server`).

```ts
import { Ed25519Verifier, KVStorageHttpServer, KVStorageInMemory } from '@harvmaster/storage'

const server = new KVStorageHttpServer(new KVStorageInMemory(), {
  // Requests must be signed, and a user may only touch keys under their public key.
  auth: { verifiers: [new Ed25519Verifier()] },
  basePath: '/kv',
  allowOrigins: ['https://app.example.com'], // CORS, if the app is on another origin
})

Deno.serve((request) => server.handle(request))
```

`auth: 'none'` turns authentication off (local development only). The server keeps its data in
whichever KV storage it wraps, and executes exactly the writes clients send (last write wins).

### Client

```ts
import { Ed25519Signer, KVStorageComposite, KVStorageEncrypted, KVStorageHttp, KVStorageInMemory, KVStorageIndexedDB } from '@harvmaster/storage'

// The user's identity: keep the private key (e.g. in IndexedDB or the OS keychain).
const signer = await Ed25519Signer.generate() // or Ed25519Signer.fromPrivateKey(saved)
const saved = await signer.exportPrivateKey()

// The encryption key: generate one and store it like the private key, or derive it from a password
// with a random salt created once and stored with the user's data (the salt isn't secret).
const salt = crypto.getRandomValues(new Uint8Array(16))
const key = await KVStorageEncrypted.deriveKey('correct horse battery staple', salt)

const indexedDb = new KVStorageIndexedDB({ name: 'my-app' })
const db = new KVStorageComposite({
  source: new KVStorageEncrypted(new KVStorageHttp({ url: 'https://api.example.com/kv', signer }), { key }),
  replicas: [new KVStorageInMemory(), indexedDb], // fastest first; the last one persists
  queue: indexedDb.child('~sync'), // offline writes, kept across reloads (outside `prefix`)
  prefix: signer.publicKey, // only this user's data is synced
})

const me = db.child(signer.publicKey)
await me.child('notes').set('1', { text: 'Works offline' }) // local immediately, then sent
await me.child('notes').get('1') // read locally

db.on('change', ({ changes, origin }) => {
  if (origin === 'remote') render(changes) // made on another device
})
```

What the composite does:

- **Reads** come from the first replica once the first sync has completed; after a reload, straight
  from IndexedDB, even offline.
- **Writes** go to every replica, then to the server. If the server is unreachable, they're queued
  and replayed in order later; `db.queued` is the number waiting. A write the server refuses (e.g. 403) is undone locally and rejected.
- **Changes from other devices** arrive over SSE and are applied, and announced as `change`
  events, only where they differ from local values.
- **`db.sync()`** replays queued writes and catches up with the server. It runs on start, whenever
  the live connection is restored, and periodically while offline.
- `onError` receives errors with no caller to reject, e.g. a queued write the server later refused
  (it's dropped, and the server's value restored).
- `db.close()` stops syncing and closes the live connection.

Documents and events work on top of the composite like any other KV storage:

```ts
type Todo = { title: string; done: boolean }
const todos = new DocumentStorageKV<{ children: { todos: Many<Todo> } }>(me).child('todos')
await todos.insert({ title: 'Ship it', done: false })
for await (const { data } of todos.find({ done: false })) console.log(data.title)
```

### Encryption

`KVStorageEncrypted` wraps any KV storage and AES-GCM encrypts each value with a fresh IV; the
key a value is stored under is authenticated too, so the server can't move values around.

- **Keys are not encrypted**: the server needs them for authorization and ordering. Don't put
  secrets in keys. Value sizes and write times are visible as well.
- `generateKey()`, `exportKey(key)` / `importKey(raw)`, or `deriveKey(password, salt)` (PBKDF2;
  store the random salt alongside the data, it isn't secret).
- Losing the key means losing the data. There is no recovery.

### Signed requests

Every request is signed with the user's key: `X-Public-Key`, `X-Timestamp`, `X-Nonce` (random
per request), `X-Signature` and `X-Signature-Algorithm` headers, over
`${timestamp}:${nonce}:${path}:${body}`. The server checks the signature, a 60 s timestamp
window and that the nonce wasn't used before (replay protection), then that every key is the
public key or under it. Ed25519 is built in (WebCrypto); other algorithms plug in by implementing
`RequestSigner` / `SignatureVerifier`.

### Live updates without the composite

```ts
const http = new KVStorageHttp({ url: 'https://api.example.com/kv', signer })
http.on('change', ({ changes }) => console.log(changes))
http.on('resync', () => reload()) // reconnected: changes may have been missed
await http.child(signer.publicKey).child('notes').subscribe()
```

Each `KVStorageHttp` instance keeps one SSE connection for all its subscriptions. Browsers allow
only 6 HTTP/1.1 connections per host, so share one client per server (or serve over HTTP/2).

## Writing your own storage

Implement the primitives of the base class for a layer; everything else is inherited. For KV:

```ts
import { BaseKVStorage, type KVChange, type ScanOptions } from '@harvmaster/storage'

class MyKVStorage extends BaseKVStorage {
  async readMany(keys: string[]): Promise<unknown[]> {
    /* values in order, undefined for missing */
  }
  async *scan(options?: ScanOptions): AsyncGenerator<[string, unknown]> {
    /* key-ordered entries within { gte, lt }, honouring limit/reverse/values */
  }
  protected async write(changes: KVChange[]): Promise<void> {
    /* set each key to its value, or delete it when value is undefined */
  }
}
```

The repository's conformance suite (`test/kv-storage/kv-storage.conformance.ts`) checks the
behaviour every KV storage must have.

## Development

```sh
npm test              # typecheck + Node tests
npm run test:browser  # Chromium tests (real IndexedDB, WebCrypto, CORS, SSE)
npm run bench
npm run format
```

CI runs formatting, typechecking, Node 22/24/26, Chromium and the build on every push, and then
publishes: `main` → `latest`, `next` → `next`, other branches → `dev-<branch>`.