Testing lots of things

docs/api.md

@@ -0,0 +1,271 @@
+# XO Sync Server API
+
+This server stores encrypted client data as append-only events. The server does
+not interpret event payloads; `data` is accepted and returned as a base64 string.
+
+## Data Model
+
+Stored events have this shape:
+
+```ts
+type StorageEvent = {
+  storageIdentifier: string;
+  data: string;      // base64 encoded encrypted payload
+  hash: string;      // base64 sha256 hash of decoded data bytes
+  created_at: number; // unix timestamp in milliseconds
+};
+```
+
+`storageIdentifier` identifies the logical append-only storage feed. It is also
+the realtime subscription topic.
+
+## Errors
+
+Validation errors return HTTP `400` with:
+
+```json
+{
+  "statusCode": 400,
+  "error": "Validation Error",
+  "details": [
+    {
+      "path": "storageIdentifier",
+      "message": "Required"
+    }
+  ]
+}
+```
+
+Unexpected errors return HTTP `500`:
+
+```json
+{
+  "error": "Internal Server Error"
+}
+```
+
+## Stats / Bloom Negotiation
+
+Use these endpoints before filtered reads to learn the server's event count,
+the inverse-bloom-filter size the client should use, and the server's current
+filter fingerprint.
+
+### `GET /stats`
+
+Alias: `GET /bloom`
+
+Query parameters:
+
+```ts
+storageIdentifier: string // required; can be comma-separated for batching
+```
+
+Example:
+
+```http
+GET /stats?storageIdentifier=/wallet/a,/wallet/b
+```
+
+Response:
+
+```json
+{
+  "items": [
+    {
+      "storageIdentifier": "/wallet/a",
+      "count": 42,
+      "negotiatedBloomFilterSize": 256,
+      "bloomFilter": "BASE64_SERVER_FILTER",
+      "bloomFilterHash": "BASE64_SHA256_OF_SERVER_FILTER",
+      "cached": false
+    },
+    {
+      "storageIdentifier": "/wallet/b",
+      "count": 1000,
+      "negotiatedBloomFilterSize": 1000,
+      "bloomFilter": "BASE64_SERVER_FILTER",
+      "bloomFilterHash": "BASE64_SHA256_OF_SERVER_FILTER",
+      "cached": true
+    }
+  ]
+}
+```
+
+`negotiatedBloomFilterSize` is currently `max(count, 256)`. Clients should build
+their inverse bloom filter using this size before calling `/data` with a filter.
+`count` is not a sync check; two peers can have the same count but different
+events. Use `bloomFilter` or `bloomFilterHash` as the server-side set
+fingerprint.
+
+### `POST /stats`
+
+Alias: `POST /bloom`
+
+Request body:
+
+```json
+{
+  "storageIdentifiers": ["/wallet/a", "/wallet/b"]
+}
+```
+
+Response is the same as `GET /stats`.
+
+## Read Data
+
+### `GET /data`
+
+Reads events for one storage identifier.
+
+Query parameters:
+
+```ts
+storageIdentifier: string; // required
+bloomFilter?: string;      // optional base64 serialized inverse bloom filter
+count?: number;            // required when bloomFilter is supplied
+```
+
+Without a filter, the server returns all events:
+
+```http
+GET /data?storageIdentifier=/wallet/a
+```
+
+Response:
+
+```json
+[
+  {
+    "storageIdentifier": "/wallet/a",
+    "data": "BASE64_PAYLOAD",
+    "hash": "BASE64_SHA256_HASH",
+    "created_at": 1760000000000
+  }
+]
+```
+
+With a compatible filter, the server returns only events that appear missing from
+the client:
+
+```http
+GET /data?storageIdentifier=/wallet/a&bloomFilter=BASE64_FILTER&count=256
+```
+
+If `count` does not match the current negotiated size, the server falls back to
+returning the full event list for correctness.
+
+If the supplied filter matches the server's current filter, the response is an
+empty list because the client already has all current server events. If the
+filters differ, the server builds or reuses its current filter, computes the
+difference, then iterates storage rows and returns only rows whose hashes appear
+missing from the client filter.
+
+## Append Data
+
+### `POST /data`
+
+Appends one event to a storage feed.
+
+Request body:
+
+```json
+{
+  "storageIdentifier": "/wallet/a",
+  "data": "BASE64_PAYLOAD"
+}
+```
+
+Response:
+
+```json
+{
+  "storageIdentifier": "/wallet/a",
+  "data": "BASE64_PAYLOAD",
+  "hash": "BASE64_SHA256_HASH",
+  "created_at": 1760000000000
+}
+```
+
+Side effects:
+
+- Invalidates the cached bloom filter for `storageIdentifier`.
+- Broadcasts a realtime `storage-event` to subscribers of that storage feed.
+
+## Stream Data
+
+### `GET /data/stream`
+
+Opens a Server-Sent Events stream for one storage identifier.
+
+Query parameters are the same as `GET /data`:
+
+```ts
+storageIdentifier: string;
+bloomFilter?: string;
+count?: number;
+```
+
+Connection behavior:
+
+1. The server opens an SSE connection.
+2. The server subscribes the connection to `storage:${storageIdentifier}`.
+3. The server calls the same handler as `GET /data`.
+4. If the handler returns data, the server sends it as the initial `sync` event.
+5. The connection remains open for live `storage-event` broadcasts.
+
+Initial sync event:
+
+```txt
+event: sync
+data: [{"storageIdentifier":"/wallet/a","data":"...","hash":"...","created_at":1760000000000}]
+```
+
+Live append event:
+
+```txt
+event: storage-event
+data: {"storageIdentifier":"/wallet/a","data":"...","hash":"...","created_at":1760000001000}
+```
+
+SSE also sends:
+
+```txt
+retry: 3000
+```
+
+Reconnect behavior uses the same sync mechanism as the first connection. Clients
+should reconnect with a fresh local inverse bloom filter:
+
+```http
+GET /data/stream?storageIdentifier=/wallet/a&bloomFilter=BASE64_FILTER&count=256
+```
+
+The server sends any missing persistent events as the initial `sync` event, then
+continues with live `storage-event` broadcasts. The server does not use
+`Last-Event-ID` for SSE replay.
+
+## Recommended Sync Flow
+
+1. Call `POST /stats` with all storage identifiers the client wants to sync.
+2. For each storage identifier, compare the server `bloomFilterHash` with the
+   client's local filter hash if available.
+3. If filters differ, build a local inverse bloom filter using
+   `negotiatedBloomFilterSize`.
+4. Call `GET /data` or `GET /data/stream` with the client's `bloomFilter` and
+   `count`.
+5. Apply returned server events locally.
+6. Compare the server `bloomFilter` from `/stats` against local events to find
+   local events the server is missing, then upload those with `POST /data`.
+7. Keep `/data/stream` open if realtime updates are needed.
+8. On reconnect, repeat the stats and bloom-filtered stream request.
+
+## Notes
+
+- Storage is append-only. There is no update or delete endpoint.
+- Payloads are opaque to the server.
+- `hash` is computed by the server from decoded `data` bytes using SHA-256.
+- `/bloom` currently exists as an alias for `/stats`; reconciliation happens in
+  `/data`, not `/bloom`.
+- The bloom-filter cache stores the current server filter for each
+  `storageIdentifier`; it does not store event payloads or a copy of all event
+  hashes. The cache is invalidated when new data is appended.