212 lines
7.1 KiB
Markdown
212 lines
7.1 KiB
Markdown
# EventDB Storage
|
|
|
|
A small local-first event and document database for applications that need offline writes, deterministic synchronization, and replaceable persistence without adopting a distributed database framework.
|
|
|
|
The package deliberately uses a simple model:
|
|
|
|
```text
|
|
DocumentDB
|
|
derives current documents from immutable events
|
|
↓
|
|
EventDB
|
|
identifies, verifies, deduplicates, and orders events
|
|
↓
|
|
BlobStorage<T>
|
|
transforms and stores logical values by immutable ID
|
|
```
|
|
|
|
It is designed for trusted applications with bounded event histories. Writes succeed locally, synchronization is idempotent, and replicas with the same event set converge to the same document state.
|
|
|
|
## Installation
|
|
|
|
```sh
|
|
npm install @harvmaster/eventdb-storage
|
|
```
|
|
|
|
The package is ESM-only and requires Web Crypto. Node.js 20 or newer and current browsers provide the required APIs. TypeScript applications should include the `DOM` library because the cross-platform public adapters expose standard Fetch, Web Crypto, LocalStorage, and IndexedDB types.
|
|
|
|
## Quick start
|
|
|
|
```ts
|
|
import {
|
|
DocumentDB,
|
|
EventDB,
|
|
IndexedDbBlobStorage,
|
|
createDocumentEventIdentity,
|
|
type DocumentEvent,
|
|
} from '@harvmaster/eventdb-storage';
|
|
|
|
const storage = new IndexedDbBlobStorage<DocumentEvent>('my-application');
|
|
|
|
const eventDB = await EventDB.open(storage, {
|
|
identify: createDocumentEventIdentity(),
|
|
});
|
|
|
|
const documentDB = await DocumentDB.open(eventDB);
|
|
|
|
const notes = documentDB.collection<{
|
|
_id: string;
|
|
title: string;
|
|
content: string;
|
|
}>('notes');
|
|
|
|
await notes.put({
|
|
_id: 'note-1',
|
|
title: 'Local first',
|
|
content: 'This write is available immediately and can replicate later.',
|
|
});
|
|
|
|
console.log(notes.get('note-1'));
|
|
|
|
await notes.delete('note-1');
|
|
|
|
// Close from the highest layer to the lowest layer.
|
|
await documentDB.close();
|
|
await eventDB.close();
|
|
await storage.close();
|
|
```
|
|
|
|
`DocumentDB.open()` performs a complete initial rebuild, so synchronous collection reads are ready when the promise resolves.
|
|
|
|
## Choosing a storage adapter
|
|
|
|
| Adapter | Intended use | Durability | Cross-context observations |
|
|
| --- | --- | --- | --- |
|
|
| `MemoryBlobStorage<T>` | Tests and temporary processing | None | Current instance only |
|
|
| `LocalStorageBlobStorage<T>` | Small browser datasets and fallbacks | Browser storage | Browser `storage` events |
|
|
| `IndexedDbBlobStorage<T>` | Normal browser applications | IndexedDB | `BroadcastChannel` when available |
|
|
| `SqliteBlobStorage<T>` | Node, Electron, native shells, and servers | Supplied SQLite database | Current instance only |
|
|
| `HttpBlobStorage<T>` | Remote persistence and SSE observations | Remote server | Server-Sent Events |
|
|
| `CompositeBlobStorage<T>` | Local-first storage with replicas | Primary adapter | Primary and replica notifications |
|
|
|
|
The SQLite adapter is driver-neutral. Supply an object implementing `SqliteDatabase`; this avoids forcing applications to install a particular native SQLite package.
|
|
|
|
## Encryption
|
|
|
|
Storage encryption is a transform and does not change event identity:
|
|
|
|
```ts
|
|
import {
|
|
LocalStorageBlobStorage,
|
|
composeStorageTransforms,
|
|
createAesGcmTransform,
|
|
createMsgPackTransform,
|
|
type DocumentEvent,
|
|
} from '@harvmaster/eventdb-storage';
|
|
|
|
const encryptionKey = await crypto.subtle.generateKey(
|
|
{ name: 'AES-GCM', length: 256 },
|
|
false,
|
|
['encrypt', 'decrypt'],
|
|
);
|
|
|
|
const transform = composeStorageTransforms(
|
|
createMsgPackTransform<DocumentEvent>(),
|
|
createAesGcmTransform(encryptionKey),
|
|
);
|
|
|
|
const storage = new LocalStorageBlobStorage<DocumentEvent>('event-db', {
|
|
transform,
|
|
});
|
|
```
|
|
|
|
AES-GCM uses a fresh random IV for every encoding. Two physical ciphertexts can therefore differ while representing the same logical event ID. Keep the encryption key and transform configuration available for the lifetime of the stored data.
|
|
|
|
## Local and remote replication
|
|
|
|
```ts
|
|
import {
|
|
CompositeBlobStorage,
|
|
HttpBlobStorage,
|
|
IndexedDbBlobStorage,
|
|
type DocumentEvent,
|
|
} from '@harvmaster/eventdb-storage';
|
|
|
|
const local = new IndexedDbBlobStorage<DocumentEvent>('event-db');
|
|
const remote = new HttpBlobStorage<DocumentEvent>('https://example.com/storage');
|
|
|
|
const storage = new CompositeBlobStorage({
|
|
primary: local,
|
|
replicas: [remote],
|
|
});
|
|
```
|
|
|
|
Normal reads use the primary. Writes persist to the primary before replication is attempted, so remote outages do not invalidate successful local writes. Composite storage performs a full immutable-ID reconciliation when constructed; `synchronize()` can also be called explicitly after connectivity returns.
|
|
|
|
### HTTP protocol
|
|
|
|
`HttpBlobStorage` expects these endpoints beneath its base URL:
|
|
|
|
- `GET /keys?prefix=...` returns a JSON array of IDs.
|
|
- `GET /blobs/:encodedId` returns transformed binary data or `404`.
|
|
- `POST /blobs/:encodedId` creates the transformed binary value and treats an existing ID as success.
|
|
- `GET /changes` returns Server-Sent Events whose JSON `data` is `{ "id": string, "data": base64 }`.
|
|
|
|
The server can remain blind to logical values and encryption keys. EventDB verifies every decoded event against its expected ID.
|
|
|
|
## Consistency model
|
|
|
|
The package guarantees:
|
|
|
|
- immutable, idempotent event storage;
|
|
- immediate local read-your-writes behavior;
|
|
- deterministic ordering by `(timestamp, eventId)`;
|
|
- convergence when replicas receive the same valid event set;
|
|
- complete recovery from authoritative events after notifications are missed;
|
|
- whole-document replacement and deletion.
|
|
|
|
It intentionally does not provide transactions across events, causal ordering, field-level merging, authorization, signatures, or proof that a remote returned every event. Timestamp-and-ID ordering is deterministic last-writer-wins, not a trusted global clock.
|
|
|
|
## Lifecycle and ownership
|
|
|
|
Each layer releases only resources it owns:
|
|
|
|
- `DocumentDB.close()` removes its EventDB listener.
|
|
- `EventDB.close()` removes its BlobStorage listeners.
|
|
- A leaf storage adapter closes its physical resource.
|
|
- `CompositeBlobStorage.close()` owns and closes its primary and replica adapters.
|
|
|
|
All `close()` methods are idempotent. Applications should stop document activity first and close dependencies in reverse construction order.
|
|
|
|
## Custom logical events
|
|
|
|
DocumentDB uses its built-in canonical identity function, but EventDB can store any event extending `BaseEventData`:
|
|
|
|
```ts
|
|
import {
|
|
CanonicalJsonEventCodec,
|
|
EventDB,
|
|
MemoryBlobStorage,
|
|
createEventIdentity,
|
|
type BaseEventData,
|
|
} from '@harvmaster/eventdb-storage';
|
|
|
|
interface CounterIncremented extends BaseEventData {
|
|
type: 'counter.incremented';
|
|
amount: number;
|
|
}
|
|
|
|
const codec = new CanonicalJsonEventCodec<CounterIncremented>();
|
|
const identify = createEventIdentity<CounterIncremented>(
|
|
(event) => codec.encodeCanonical(event),
|
|
);
|
|
const storage = new MemoryBlobStorage<CounterIncremented>();
|
|
const events = await EventDB.open(storage, { identify });
|
|
```
|
|
|
|
Canonical identity supports plain objects, arrays, finite numbers, strings, booleans, `null`, `Uint8Array`, `Date`, and `bigint`. Functions, symbols, sparse arrays, cyclic references, accessors, and non-plain object instances are rejected.
|
|
|
|
## Development
|
|
|
|
```sh
|
|
npm test
|
|
npm run typecheck
|
|
npm run build
|
|
```
|
|
|
|
The compiled package and declarations are written to `dist/`.
|
|
|
|
## License
|
|
|
|
ISC. See [LICENSE](./LICENSE).
|