PotentialStorage/README.md

@harvmaster/storage

Layered TypeScript storage that works the same in the browser and on the server:

• KV storage: ordered keys and values, navigated as a tree (db.child('users').get('1')). In memory, IndexedDB, or over HTTP, with live updates over Server-Sent Events.
• Event storage: append-only streams on top of any KV storage.
• Document storage: JSON documents with queries, updates ($set, $inc, …) and indexes, on top of any KV storage, or event-sourced.
• Sync: local replicas that work offline and stay up to date with a server (KVStorageComposite), end-to-end encryption (KVStorageEncrypted) and signed requests where the user's public key is their id.

No runtime dependencies. Node 22+ and current browsers. Values are JSON plus bigint and Uint8Array.

Status: experimental (0.x). The API may change between minor versions. See DESIGN.md for how everything works and its known limitations.

Install

The package is published to a private registry. Add to your project's .npmrc:

@harvmaster:registry=https://verdaccio.harvmaster.com/

npm install @harvmaster/storage          # latest release
npm install @harvmaster/storage@next     # `next` branch
npm install @harvmaster/storage@dev-sync # a development branch (`dev-<branch>`)

KV storage

Every store is a tree of nodes. child(key) returns a view rooted deeper (no I/O); methods take an optional key to act on a child instead of the node itself.

import { KVStorageInMemory } from '@harvmaster/storage'

const db = new KVStorageInMemory()
const users = db.child('users')

await users.set('1', { name: 'Ada' })
await users.child('1').get() // { name: 'Ada' }
await users.get('1') // the same
await users.setMany([
  ['2', { name: 'Bob' }],
  ['3', { name: 'Cy' }],
])
await users.getMany(['1', 'missing']) // [{ name: 'Ada' }, undefined]

for await (const key of users.keys({ limit: 10 })) console.log(key) // '1', '2', '3'
for await (const [key, user] of users.entries({ reverse: true })) console.log(key, user)

await users.delete('3')
await users.deleteTree() // users and everything under it

A node's own value and its children are independent, so an object can also be stored field per key:

const settings = db.child('settings')
await settings.setFields({ theme: 'dark', fontSize: 14 })
await settings.get('theme') // 'dark'
await settings.getFields() // { fontSize: 14, theme: 'dark' }

Keys sort as strings ('10' before '9'), and any string or number is a valid key: special characters are escaped. Every write emits a change event:

db.on('change', ({ changes, origin }) => console.log(origin, changes)) // 'local', [{ key: 'users/1', value: … }]

Types

Describe the tree once to get typed keys and values:

import { KVStorageInMemory, type Many, type Fields } from '@harvmaster/storage'

type User = { name: string }
type Schema = { children: { users: Many<User>; settings: { children: Fields<{ theme: string; fontSize: number }> } } }

const typed = new KVStorageInMemory<Schema>()
await typed.child('users').set('1', { name: 'Ada' }) // checked
const theme = await typed.child('settings').get('theme') // string | undefined

IndexedDB

import { KVStorageIndexedDB } from '@harvmaster/storage'

const local = new KVStorageIndexedDB({ name: 'my-app' })

Event storage

Append-only streams. Every node is a stream (its own events) and a parent of other streams. Appends never conflict: concurrent writers each add their own events, and every reader sees all of them in the same (time) order.

import { EventStorageKV, KVStorageInMemory } from '@harvmaster/storage'

const events = new EventStorageKV(new KVStorageInMemory())
const orders = events.child('orders')

const [id] = await orders.append('o1', [{ type: 'created', total: 30 }])
await orders.child('o1').append([{ type: 'paid' }])

for await (const event of orders.read('o1')) console.log(event.id, event.at, event.data)
for await (const event of orders.read('o1', { after: id })) console.log(event.data) // { type: 'paid' }
for await (const key of orders.keys()) console.log(key) // 'o1'

Event ids are <time>_<hash of the time and content>, so writing the same event again (e.g. a retried request) can't duplicate it.

Document storage

import { DocumentStorageKV, KVStorageInMemory } from '@harvmaster/storage'

const docs = new DocumentStorageKV(new KVStorageInMemory())
const people = docs.child('people')

const id = await people.insert({ name: 'Ada', age: 36, tags: ['math'] })
await people.set('grace', { name: 'Grace', age: 45, tags: ['navy', 'cobol'] })
await people.update(id, { $inc: { age: 1 }, $push: { tags: 'engines' } })

for await (const { key, data } of people.find({ age: { $gte: 37 } }, { sort: { age: -1 }, limit: 10 })) console.log(key, data)
await people.findOne({ tags: 'cobol' }) // { key: 'grace', data: { … } }
await people.delete('grace') // true

Filters: { field: value } (dot paths; arrays match any element), $eq $gt $gte $lt $lte $in. Updates: $set $unset $inc $push $pull $addToSet ($each for push/addToSet).

• DocumentStorageKV stores each document as one KV value, so it works over any KV storage (including a synced one, below). Queries scan the collection.
• DocumentStorageInMemory({ indexes: { people: ['age'] } }) keeps equality indexes.
• EventSourcedDocumentStorage(events, projections) records every change as an event (with history(key) and getAt(key, eventId)), and keeps the current state in any document store.

Sync, offline and encryption

The pieces below combine into an offline-first app whose server only stores ciphertext.

Server

KVStorageHttpServer serves any KV storage over HTTP. handle(request) takes a Fetch Request and returns a Response, so it works with Deno.serve, Bun.serve, Cloudflare Workers, Hono, and so on (in Node, through an adapter such as @hono/node-server).

import { Ed25519Verifier, KVStorageHttpServer, KVStorageInMemory } from '@harvmaster/storage'

const server = new KVStorageHttpServer(new KVStorageInMemory(), {
  // Requests must be signed, and a user may only touch keys under their public key.
  auth: { verifiers: [new Ed25519Verifier()] },
  basePath: '/kv',
  allowOrigins: ['https://app.example.com'], // CORS, if the app is on another origin
})

Deno.serve((request) => server.handle(request))

auth: 'none' turns authentication off (local development only). The server keeps its data in whichever KV storage it wraps, and executes exactly the writes clients send (last write wins).

Client

import { Ed25519Signer, KVStorageComposite, KVStorageEncrypted, KVStorageHttp, KVStorageInMemory, KVStorageIndexedDB } from '@harvmaster/storage'

// The user's identity: keep the private key (e.g. in IndexedDB or the OS keychain).
const signer = await Ed25519Signer.generate() // or Ed25519Signer.fromPrivateKey(saved)
const saved = await signer.exportPrivateKey()

// The encryption key: generate one and store it like the private key, or derive it from a password
// with a random salt created once and stored with the user's data (the salt isn't secret).
const salt = crypto.getRandomValues(new Uint8Array(16))
const key = await KVStorageEncrypted.deriveKey('correct horse battery staple', salt)

const indexedDb = new KVStorageIndexedDB({ name: 'my-app' })
const db = new KVStorageComposite({
  source: new KVStorageEncrypted(new KVStorageHttp({ url: 'https://api.example.com/kv', signer }), { key }),
  replicas: [new KVStorageInMemory(), indexedDb], // fastest first; the last one persists
  queue: indexedDb.child('~sync'), // offline writes, kept across reloads (outside `prefix`)
  prefix: signer.publicKey, // only this user's data is synced
})

const me = db.child(signer.publicKey)
await me.child('notes').set('1', { text: 'Works offline' }) // local immediately, then sent
await me.child('notes').get('1') // read locally

db.on('change', ({ changes, origin }) => {
  if (origin === 'remote') render(changes) // made on another device
})

What the composite does:

• Reads come from the first replica once the first sync has completed; after a reload, straight from IndexedDB, even offline.
• Writes go to every replica, then to the server. If the server is unreachable, they're queued and replayed in order later; db.queued is the number waiting. A write the server refuses (e.g. 403) is undone locally and rejected.
• Changes from other devices arrive over SSE and are applied, and announced as change events, only where they differ from local values.
• db.sync() replays queued writes and catches up with the server. It runs on start, whenever the live connection is restored, and periodically while offline.
• onError receives errors with no caller to reject, e.g. a queued write the server later refused (it's dropped, and the server's value restored).
• db.close() stops syncing and closes the live connection.

Documents and events work on top of the composite like any other KV storage:

type Todo = { title: string; done: boolean }
const todos = new DocumentStorageKV<{ children: { todos: Many<Todo> } }>(me).child('todos')
await todos.insert({ title: 'Ship it', done: false })
for await (const { data } of todos.find({ done: false })) console.log(data.title)

Encryption

KVStorageEncrypted wraps any KV storage and AES-GCM encrypts each value with a fresh IV; the key a value is stored under is authenticated too, so the server can't move values around.

• Keys are not encrypted: the server needs them for authorization and ordering. Don't put secrets in keys. Value sizes and write times are visible as well.
• generateKey(), exportKey(key) / importKey(raw), or deriveKey(password, salt) (PBKDF2; store the random salt alongside the data, it isn't secret).
• Losing the key means losing the data. There is no recovery.

Signed requests

Every request is signed with the user's key: X-Public-Key, X-Timestamp, X-Nonce (random per request), X-Signature and X-Signature-Algorithm headers, over ${timestamp}:${nonce}:${path}:${body}. The server checks the signature, a 60 s timestamp window and that the nonce wasn't used before (replay protection), then that every key is the public key or under it. Ed25519 is built in (WebCrypto); other algorithms plug in by implementing RequestSigner / SignatureVerifier.

Live updates without the composite

const http = new KVStorageHttp({ url: 'https://api.example.com/kv', signer })
http.on('change', ({ changes }) => console.log(changes))
http.on('resync', () => reload()) // reconnected: changes may have been missed
await http.child(signer.publicKey).child('notes').subscribe()

Each KVStorageHttp instance keeps one SSE connection for all its subscriptions. Browsers allow only 6 HTTP/1.1 connections per host, so share one client per server (or serve over HTTP/2).

Writing your own storage

Implement the primitives of the base class for a layer; everything else is inherited. For KV:

import { BaseKVStorage, type KVChange, type ScanOptions } from '@harvmaster/storage'

class MyKVStorage extends BaseKVStorage {
  async readMany(keys: string[]): Promise<unknown[]> {
    /* values in order, undefined for missing */
  }
  async *scan(options?: ScanOptions): AsyncGenerator<[string, unknown]> {
    /* key-ordered entries within { gte, lt }, honouring limit/reverse/values */
  }
  protected async write(changes: KVChange[]): Promise<void> {
    /* set each key to its value, or delete it when value is undefined */
  }
}

The repository's conformance suite (test/kv-storage/kv-storage.conformance.ts) checks the behaviour every KV storage must have.

Development

npm test              # typecheck + Node tests
npm run test:browser  # Chromium tests (real IndexedDB, WebCrypto, CORS, SSE)
npm run bench
npm run format

CI runs formatting, typechecking, Node 22/24/26, Chromium and the build on every push, and then publishes: main → latest, next → next, other branches → dev-<branch>.