M9: benchmarks, chunked sorted map and JSON clone for the in-memory driver
Profiling showed document updates dominated by O(n) sorted-array splices; the chunked map makes inserts O(log n + chunk). Baselines recorded in DESIGN.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017Qxbkp4tA7G5SCWjzWuxmE
This commit is contained in:
@@ -589,8 +589,58 @@ Decisions taken where the brief was silent. Each is the simplest option consiste
|
|||||||
socket, and against a minimal server driver with discovered capabilities. A smoke test runs KV
|
socket, and against a minimal server driver with discovered capabilities. A smoke test runs KV
|
||||||
indexes, events and unique-indexed documents over HTTP.
|
indexes, events and unique-indexed documents over HTTP.
|
||||||
|
|
||||||
|
### M9 — Benchmarks and in-memory driver performance
|
||||||
|
|
||||||
|
- **The in-memory driver's sorted array was replaced by `ChunkedSortedMap`** (sorted chunks of ≤1024
|
||||||
|
keys, split in half on overflow; values in a `Map`). Profiling showed 59% of document-update time
|
||||||
|
in the sorted array's O(n) `splice` once the store held ~70k keys (each write adds commit, index
|
||||||
|
and claim keys). Inserts and deletes are now O(log n + 1024). Iteration is O(1) per step and
|
||||||
|
re-seeks after a mutation, which replaces the old version-counter revalidation in the driver.
|
||||||
|
The structure stays behind the `SortedMap` interface; a B-tree remains possible.
|
||||||
|
- **Value cloning uses a hand-written JSON deep copy** instead of `structuredClone` (about 10× cheaper
|
||||||
|
per value), in the in-memory driver and the document layer. Values are JSON by contract (§10).
|
||||||
|
|
||||||
<!-- decisions appended per milestone -->
|
<!-- decisions appended per milestone -->
|
||||||
|
|
||||||
## 13. Benchmark baselines
|
## 13. Benchmark baselines
|
||||||
|
|
||||||
_Filled in at milestone 9._
|
`npm run bench` (Vitest bench), in-memory driver with default cloning, Node 22.22, Linux x64 cloud
|
||||||
|
container (shared CPU, so expect ±10–20% run-to-run). Mean time per operation.
|
||||||
|
|
||||||
|
**KV** (`bench/kv.bench.ts`): values are small objects.
|
||||||
|
|
||||||
|
| Operation | 10k keys | 100k keys |
|
||||||
|
|---|---|---|
|
||||||
|
| `get` | 1.2 µs | 1.6 µs |
|
||||||
|
| `set` (overwrite) | 1.7 µs | 2.0 µs |
|
||||||
|
| `getMany` × 100 keys | 21 µs | 22 µs |
|
||||||
|
|
||||||
|
**`keys()`** (`bench/keys.bench.ts`): node with 1 000 children, each with 100 descendants (101k keys).
|
||||||
|
|
||||||
|
| Operation | Mean |
|
||||||
|
|---|---|
|
||||||
|
| `keys()` skip-scan (1 000 seeks) | 2.9 ms |
|
||||||
|
| Reference: full subtree scan | 36 ms |
|
||||||
|
|
||||||
|
**Events** (`bench/events.bench.ts`): single-event commits.
|
||||||
|
|
||||||
|
| Operation | Mean |
|
||||||
|
|---|---|
|
||||||
|
| `append` 1 event (warm head cache) | 6.3 µs |
|
||||||
|
| `read` 100 events from version 5 000 of 10 000 | 174 µs |
|
||||||
|
| `read` full 10 000-event stream | 21 ms |
|
||||||
|
| `version()` (reverse scan, `limit: 1`) | 3.9 µs |
|
||||||
|
|
||||||
|
**Documents** (`bench/documents.bench.ts`): 10 000 docs, indexes on `age` and unique `email`.
|
||||||
|
|
||||||
|
| Operation | Mean |
|
||||||
|
|---|---|
|
||||||
|
| `get` | 2.6 µs |
|
||||||
|
| `update` (`$inc`, indexed node: projection read + append + batch) | 22 µs |
|
||||||
|
| `findOne` by unique index (`email`) | 16 µs |
|
||||||
|
| `find` indexed (`age = k`, ~100 hits) | 0.57 ms |
|
||||||
|
| `find` full scan (`city = k`, ~200 hits of 10k) | 28 ms |
|
||||||
|
|
||||||
|
For comparison, the first run with the sorted-array map and `structuredClone` measured: KV `get`
|
||||||
|
3.7 µs, `getMany`×100 186 µs, `append` 34 µs, document `update` 110 µs, full scan 57 ms, and
|
||||||
|
full stream read 62 ms.
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
import { bench, describe } from 'vitest'
|
||||||
|
import { InMemoryKVDriver } from '../src/kv/memory.js'
|
||||||
|
import { KVStorage } from '../src/kv/node.js'
|
||||||
|
import { KVEventStorage } from '../src/events/kv.js'
|
||||||
|
import { EventSourcedDocumentStorage } from '../src/documents/event-sourced.js'
|
||||||
|
|
||||||
|
const N = 10_000
|
||||||
|
const docs = new EventSourcedDocumentStorage(new KVEventStorage(new KVStorage(new InMemoryKVDriver())), {
|
||||||
|
indexes: { users: [{ fields: ['age'] }, { fields: ['email'], unique: true }] },
|
||||||
|
warnOnFullScan: false,
|
||||||
|
})
|
||||||
|
const users = docs.child('users')
|
||||||
|
for (let i = 0; i < N; i++) await users.set(`u${i}`, { name: `user ${i}`, email: `u${i}@x.io`, age: i % 100, city: `c${i % 50}` })
|
||||||
|
let i = 0
|
||||||
|
|
||||||
|
describe(`documents @ ${N / 1000}k docs`, () => {
|
||||||
|
bench('get', async () => {
|
||||||
|
await users.get(`u${i++ % N}`)
|
||||||
|
})
|
||||||
|
bench('update ($inc, indexed node)', async () => {
|
||||||
|
await users.update(`u${i++ % N}`, { $inc: { logins: 1 } })
|
||||||
|
})
|
||||||
|
bench('findOne by unique index (email)', async () => {
|
||||||
|
await users.findOne({ email: `u${i++ % N}@x.io` })
|
||||||
|
})
|
||||||
|
bench('find indexed (age = k, ~100 hits)', async () => {
|
||||||
|
for await (const _ of users.find({ age: i++ % 100 })) {
|
||||||
|
}
|
||||||
|
})
|
||||||
|
bench('find full scan (city = k, ~200 hits)', async () => {
|
||||||
|
for await (const _ of users.find({ city: `c${i++ % 50}` })) {
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
import { bench, describe } from 'vitest'
|
||||||
|
import { InMemoryKVDriver } from '../src/kv/memory.js'
|
||||||
|
import { KVStorage } from '../src/kv/node.js'
|
||||||
|
import { KVEventStorage } from '../src/events/kv.js'
|
||||||
|
|
||||||
|
const events = new KVEventStorage(new KVStorage(new InMemoryKVDriver()))
|
||||||
|
const streams = events.child('streams')
|
||||||
|
const long = streams.child('long')
|
||||||
|
for (let i = 0; i < 10_000; i++) await long.append([{ type: 'tick', i }])
|
||||||
|
let s = 0
|
||||||
|
|
||||||
|
describe('events', () => {
|
||||||
|
bench('append 1 event (warm head cache)', async () => {
|
||||||
|
await streams.append(`s${s++ % 1000}`, [{ type: 'e' }])
|
||||||
|
})
|
||||||
|
bench('read 100 events from version 5000 of 10k', async () => {
|
||||||
|
for await (const _ of long.read({ from: 5000, limit: 100 })) {
|
||||||
|
}
|
||||||
|
})
|
||||||
|
bench('read full 10k-event stream', async () => {
|
||||||
|
for await (const _ of long.read()) {
|
||||||
|
}
|
||||||
|
})
|
||||||
|
bench('version() (cold head lookup)', async () => {
|
||||||
|
await long.version()
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
import { bench, describe } from 'vitest'
|
||||||
|
import { InMemoryKVDriver } from '../src/kv/memory.js'
|
||||||
|
import { KVStorage } from '../src/kv/node.js'
|
||||||
|
|
||||||
|
// 1k children, each with 100 descendants (100k keys under the node).
|
||||||
|
const db = new KVStorage(new InMemoryKVDriver({ clone: false }))
|
||||||
|
const node = db.child('n')
|
||||||
|
const writes: [string, number][] = []
|
||||||
|
for (let c = 0; c < 1000; c++) {
|
||||||
|
writes.push([`n/c${c}`, c])
|
||||||
|
for (let d = 0; d < 100; d++) writes.push([`n/c${c}/d${d}`, d])
|
||||||
|
}
|
||||||
|
await db.driver.setMany(writes)
|
||||||
|
|
||||||
|
describe('keys() on 1k children x 100 descendants', () => {
|
||||||
|
bench('keys() skip-scan', async () => {
|
||||||
|
let n = 0
|
||||||
|
for await (const _ of node.keys()) n++
|
||||||
|
if (n !== 1000) throw new Error(`expected 1000, got ${n}`)
|
||||||
|
})
|
||||||
|
bench('naive full subtree scan (reference)', async () => {
|
||||||
|
let n = 0
|
||||||
|
for await (const _ of db.driver.scan({ gte: 'n/', lt: 'n0' })) n++
|
||||||
|
if (n !== 101_000) throw new Error(`expected 101000, got ${n}`)
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import { bench, describe } from 'vitest'
|
||||||
|
import { InMemoryKVDriver } from '../src/kv/memory.js'
|
||||||
|
import { KVStorage } from '../src/kv/node.js'
|
||||||
|
|
||||||
|
async function populated(n: number) {
|
||||||
|
const db = new KVStorage(new InMemoryKVDriver())
|
||||||
|
const users = db.child('users')
|
||||||
|
const batch: [string, { name: string; n: number }][] = []
|
||||||
|
for (let i = 0; i < n; i++) batch.push([`u${i}`, { name: `user ${i}`, n: i }])
|
||||||
|
await users.setMany(batch)
|
||||||
|
return users
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const n of [10_000, 100_000]) {
|
||||||
|
const users = await populated(n)
|
||||||
|
const keys = Array.from({ length: 100 }, (_, i) => `u${(i * 7919) % n}`)
|
||||||
|
let i = 0
|
||||||
|
describe(`KV @ ${n / 1000}k keys`, () => {
|
||||||
|
bench('get', async () => {
|
||||||
|
await users.get(`u${i++ % n}`)
|
||||||
|
})
|
||||||
|
bench('set (overwrite)', async () => {
|
||||||
|
await users.set(`u${i++ % n}`, { name: 'x', n: i })
|
||||||
|
})
|
||||||
|
bench('getMany x100', async () => {
|
||||||
|
await users.getMany(keys)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -67,7 +67,12 @@ export class DocIndexes {
|
|||||||
let multi = 0
|
let multi = 0
|
||||||
for (const segs of idx.segs) {
|
for (const segs of idx.segs) {
|
||||||
const found = resolvePath(state, segs)
|
const found = resolvePath(state, segs)
|
||||||
const scalars = found.length === 0 ? [null] : [...new Set(found.flatMap(indexScalars))]
|
const scalars =
|
||||||
|
found.length === 0 ? [null] : found.length === 1 && !Array.isArray(found[0]) ? indexScalars(found[0]) : [...new Set(found.flatMap(indexScalars))]
|
||||||
|
if (tuples.length === 1 && scalars.length === 1) {
|
||||||
|
tuples[0]!.push(scalars[0]!)
|
||||||
|
continue
|
||||||
|
}
|
||||||
if (scalars.length > 1 && ++multi > 1) throw new TypeError(`Index "${idx.name}": cannot index more than one array field per document`)
|
if (scalars.length > 1 && ++multi > 1) throw new TypeError(`Index "${idx.name}": cannot index more than one array field per document`)
|
||||||
tuples = tuples.flatMap((t) => scalars.map((s) => [...t, s]))
|
tuples = tuples.flatMap((t) => scalars.map((s) => [...t, s]))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -98,8 +98,13 @@ export function deepEqual(a: unknown, b: unknown): boolean {
|
|||||||
return ka.length === kb.length && ka.every((k) => Object.hasOwn(b, k) && deepEqual((a as Doc)[k], (b as Doc)[k]))
|
return ka.length === kb.length && ka.every((k) => Object.hasOwn(b, k) && deepEqual((a as Doc)[k], (b as Doc)[k]))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Deep copy of a JSON-compatible value (documents are JSON). */
|
||||||
export function clone<T>(v: T): T {
|
export function clone<T>(v: T): T {
|
||||||
return v === null || typeof v !== 'object' ? v : structuredClone(v)
|
if (v === null || typeof v !== 'object') return v
|
||||||
|
if (Array.isArray(v)) return v.map(clone) as T
|
||||||
|
const out: Doc = {}
|
||||||
|
for (const k in v) out[k] = clone((v as Doc)[k])
|
||||||
|
return out as T
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Type class used for bracketing comparisons, in index-encoding order. */
|
/** Type class used for bracketing comparisons, in index-encoding order. */
|
||||||
|
|||||||
+20
-31
@@ -1,11 +1,11 @@
|
|||||||
import { BaseKVDriver, type BatchOp, type ScanOptions } from './driver.js'
|
import { BaseKVDriver, type BatchOp, type ScanOptions } from './driver.js'
|
||||||
import { SortedArrayMap, type SortedMap } from './sorted.js'
|
import { ChunkedSortedMap, type SortedMap } from './sorted.js'
|
||||||
|
|
||||||
export interface InMemoryKVDriverOptions {
|
export interface InMemoryKVDriverOptions {
|
||||||
/**
|
/**
|
||||||
* Copy values on write and read so callers can't mutate stored state through a shared
|
* Deep-copy values on write and read so callers can't mutate stored state through a shared
|
||||||
* reference (matching serialising drivers). Default `true`. Disable for speed when values
|
* reference (matching serialising drivers). Values must be JSON-compatible. Default `true`.
|
||||||
* are treated as immutable.
|
* Disable for speed when values are treated as immutable.
|
||||||
*/
|
*/
|
||||||
clone?: boolean
|
clone?: boolean
|
||||||
}
|
}
|
||||||
@@ -15,7 +15,7 @@ export interface InMemoryKVDriverOptions {
|
|||||||
* `setIfAbsent`, atomic `batch` and `deleteRange`.
|
* `setIfAbsent`, atomic `batch` and `deleteRange`.
|
||||||
*/
|
*/
|
||||||
export class InMemoryKVDriver extends BaseKVDriver {
|
export class InMemoryKVDriver extends BaseKVDriver {
|
||||||
readonly #data: SortedMap<unknown> = new SortedArrayMap()
|
readonly #data: SortedMap<unknown> = new ChunkedSortedMap()
|
||||||
readonly #clone: (v: unknown) => unknown
|
readonly #clone: (v: unknown) => unknown
|
||||||
|
|
||||||
constructor(opts: InMemoryKVDriverOptions = {}) {
|
constructor(opts: InMemoryKVDriverOptions = {}) {
|
||||||
@@ -59,37 +59,17 @@ export class InMemoryKVDriver extends BaseKVDriver {
|
|||||||
}
|
}
|
||||||
|
|
||||||
override async deleteRange(gte: string, lt: string): Promise<void> {
|
override async deleteRange(gte: string, lt: string): Promise<void> {
|
||||||
const data = this.#data
|
const doomed = [...this.#data.keys({ gte, lt })]
|
||||||
const start = data.lowerBound(gte)
|
for (const key of doomed) this.#data.delete(key)
|
||||||
const doomed: string[] = []
|
|
||||||
for (let i = start; ; i++) {
|
|
||||||
const key = data.keyAt(i)
|
|
||||||
if (key === undefined || key >= lt) break
|
|
||||||
doomed.push(key)
|
|
||||||
}
|
|
||||||
for (const key of doomed) data.delete(key)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async *scan(opts: ScanOptions): AsyncGenerator<[string, unknown]> {
|
async *scan(opts: ScanOptions): AsyncGenerator<[string, unknown]> {
|
||||||
const data = this.#data
|
|
||||||
const { gte, lt, reverse = false, values = false } = opts
|
const { gte, lt, reverse = false, values = false } = opts
|
||||||
let remaining = opts.limit ?? Infinity
|
let remaining = opts.limit ?? Infinity
|
||||||
if (remaining <= 0) return
|
if (remaining <= 0) return
|
||||||
// `last` is the last key yielded; if the map changes while we're suspended, re-seek from it.
|
// The key iterator re-seeks by itself if the map changes while we're suspended.
|
||||||
let last: string | undefined
|
for (const key of this.#data.keys({ gte, lt, reverse })) {
|
||||||
let version = data.version
|
yield [key, values ? this.#clone(this.#data.get(key)) : undefined]
|
||||||
let i = reverse ? (lt === undefined ? data.size : data.lowerBound(lt)) - 1 : gte === undefined ? 0 : data.lowerBound(gte)
|
|
||||||
for (;;) {
|
|
||||||
if (data.version !== version) {
|
|
||||||
version = data.version
|
|
||||||
if (last !== undefined) i = reverse ? data.lowerBound(last) - 1 : data.lowerBound(last, true)
|
|
||||||
}
|
|
||||||
const key = data.keyAt(i)
|
|
||||||
if (key === undefined || i < 0) return
|
|
||||||
if (reverse ? gte !== undefined && key < gte : lt !== undefined && key >= lt) return
|
|
||||||
last = key
|
|
||||||
i += reverse ? -1 : 1
|
|
||||||
yield [key, values ? this.#clone(data.get(key)) : undefined]
|
|
||||||
if (--remaining <= 0) return
|
if (--remaining <= 0) return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -99,6 +79,15 @@ function identity(v: unknown): unknown {
|
|||||||
return v
|
return v
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Deep copy of a JSON-compatible value (much faster than `structuredClone`). */
|
||||||
function cloneValue(v: unknown): unknown {
|
function cloneValue(v: unknown): unknown {
|
||||||
return v === null || typeof v !== 'object' ? v : structuredClone(v)
|
if (v === null || typeof v !== 'object') return v
|
||||||
|
if (Array.isArray(v)) {
|
||||||
|
const out = new Array<unknown>(v.length)
|
||||||
|
for (let i = 0; i < v.length; i++) out[i] = cloneValue(v[i])
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
const out: Record<string, unknown> = {}
|
||||||
|
for (const k in v) out[k] = cloneValue((v as Record<string, unknown>)[k])
|
||||||
|
return out
|
||||||
}
|
}
|
||||||
|
|||||||
+116
-35
@@ -1,33 +1,37 @@
|
|||||||
/**
|
/**
|
||||||
* Ordered string-keyed map used by the in-memory driver. Kept behind an interface so the
|
* Ordered string-keyed map used by the in-memory driver. Kept behind an interface so the
|
||||||
* sorted-array implementation can be swapped for a B-tree without touching the driver.
|
* implementation can be swapped (e.g. for a B-tree) without touching the driver.
|
||||||
*/
|
*/
|
||||||
export interface SortedMap<V> {
|
export interface SortedMap<V> {
|
||||||
readonly size: number
|
readonly size: number
|
||||||
/** Incremented on every structural change (insert or delete), for iterator revalidation. */
|
|
||||||
readonly version: number
|
|
||||||
get(key: string): V | undefined
|
get(key: string): V | undefined
|
||||||
has(key: string): boolean
|
has(key: string): boolean
|
||||||
set(key: string, value: V): void
|
set(key: string, value: V): void
|
||||||
/** Inserts many entries; may be faster than repeated `set`. */
|
/** Inserts many entries; may be faster than repeated `set`. */
|
||||||
setMany(entries: Iterable<[string, V]>): void
|
setMany(entries: Iterable<[string, V]>): void
|
||||||
delete(key: string): boolean
|
delete(key: string): boolean
|
||||||
/** Index of the first key `>= key` (or `> key` when `exclusive`). */
|
/**
|
||||||
lowerBound(key: string, exclusive?: boolean): number
|
* Keys in `[gte, lt)` in order (descending when `reverse`). The iterator stays valid across
|
||||||
keyAt(index: number): string | undefined
|
* mutations between steps: it resumes after (before) the last key it returned.
|
||||||
|
*/
|
||||||
|
keys(range: { gte?: string; lt?: string; reverse?: boolean }): IterableIterator<string>
|
||||||
}
|
}
|
||||||
|
|
||||||
export class SortedArrayMap<V> implements SortedMap<V> {
|
|
||||||
#keys: string[] = []
|
/**
|
||||||
#values = new Map<string, V>()
|
* Sorted chunks of at most `maxChunk` keys (split in half on overflow) plus a `Map` for values.
|
||||||
|
* Inserts and deletes cost O(log n + maxChunk); iteration is O(1) per step while the map
|
||||||
|
* is unchanged and re-seeks in O(log n) after a mutation.
|
||||||
|
*/
|
||||||
|
export class ChunkedSortedMap<V> implements SortedMap<V> {
|
||||||
|
constructor(readonly maxChunk = 1024) {}
|
||||||
|
|
||||||
|
#chunks: string[][] = [[]]
|
||||||
|
readonly #values = new Map<string, V>()
|
||||||
#version = 0
|
#version = 0
|
||||||
|
|
||||||
get size(): number {
|
get size(): number {
|
||||||
return this.#keys.length
|
return this.#values.size
|
||||||
}
|
|
||||||
|
|
||||||
get version(): number {
|
|
||||||
return this.#version
|
|
||||||
}
|
}
|
||||||
|
|
||||||
get(key: string): V | undefined {
|
get(key: string): V | undefined {
|
||||||
@@ -39,10 +43,7 @@ export class SortedArrayMap<V> implements SortedMap<V> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
set(key: string, value: V): void {
|
set(key: string, value: V): void {
|
||||||
if (!this.#values.has(key)) {
|
if (!this.#values.has(key)) this.#insert(key)
|
||||||
this.#keys.splice(this.lowerBound(key), 0, key)
|
|
||||||
this.#version++
|
|
||||||
}
|
|
||||||
this.#values.set(key, value)
|
this.#values.set(key, value)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -53,14 +54,13 @@ export class SortedArrayMap<V> implements SortedMap<V> {
|
|||||||
this.#values.set(key, value)
|
this.#values.set(key, value)
|
||||||
}
|
}
|
||||||
if (fresh.length === 0) return
|
if (fresh.length === 0) return
|
||||||
this.#version++
|
if (fresh.length < 64 || fresh.length * 8 < this.#values.size) {
|
||||||
if (fresh.length < 32) {
|
for (const key of fresh) this.#insert(key)
|
||||||
for (const key of fresh) this.#keys.splice(this.lowerBound(key), 0, key)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
// Bulk path: sort the new keys and merge, O(n + k log k) instead of O(n·k).
|
// Bulk path: merge and re-chunk, O(n + k log k).
|
||||||
fresh.sort(cmp)
|
fresh.sort(cmp)
|
||||||
const old = this.#keys
|
const old = this.#chunks.flat()
|
||||||
const merged = new Array<string>(old.length + fresh.length)
|
const merged = new Array<string>(old.length + fresh.length)
|
||||||
let i = 0
|
let i = 0
|
||||||
let j = 0
|
let j = 0
|
||||||
@@ -68,34 +68,115 @@ export class SortedArrayMap<V> implements SortedMap<V> {
|
|||||||
while (i < old.length && j < fresh.length) merged[k++] = old[i]! < fresh[j]! ? old[i++]! : fresh[j++]!
|
while (i < old.length && j < fresh.length) merged[k++] = old[i]! < fresh[j]! ? old[i++]! : fresh[j++]!
|
||||||
while (i < old.length) merged[k++] = old[i++]!
|
while (i < old.length) merged[k++] = old[i++]!
|
||||||
while (j < fresh.length) merged[k++] = fresh[j++]!
|
while (j < fresh.length) merged[k++] = fresh[j++]!
|
||||||
this.#keys = merged
|
const half = Math.max(1, this.maxChunk >>> 1)
|
||||||
|
this.#chunks = []
|
||||||
|
for (let s = 0; s < merged.length; s += half) this.#chunks.push(merged.slice(s, s + half))
|
||||||
|
this.#version++
|
||||||
}
|
}
|
||||||
|
|
||||||
delete(key: string): boolean {
|
delete(key: string): boolean {
|
||||||
if (!this.#values.delete(key)) return false
|
if (!this.#values.delete(key)) return false
|
||||||
this.#keys.splice(this.lowerBound(key), 1)
|
const c = this.#chunkFor(key)
|
||||||
|
const chunk = this.#chunks[c]!
|
||||||
|
chunk.splice(lowerBound(chunk, key), 1)
|
||||||
|
if (chunk.length === 0 && this.#chunks.length > 1) this.#chunks.splice(c, 1)
|
||||||
this.#version++
|
this.#version++
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
lowerBound(key: string, exclusive = false): number {
|
*keys(range: { gte?: string; lt?: string; reverse?: boolean }): IterableIterator<string> {
|
||||||
const keys = this.#keys
|
const { gte, lt, reverse = false } = range
|
||||||
|
let version = this.#version
|
||||||
|
let c: number
|
||||||
|
let i: number
|
||||||
|
// Position of the first candidate.
|
||||||
|
if (reverse) {
|
||||||
|
;[c, i] = lt === undefined ? this.#end() : this.#seek(lt, false)
|
||||||
|
;[c, i] = this.#prev(c, i)
|
||||||
|
} else {
|
||||||
|
;[c, i] = gte === undefined ? [0, 0] : this.#seek(gte, false)
|
||||||
|
;[c, i] = this.#normalize(c, i)
|
||||||
|
}
|
||||||
|
for (;;) {
|
||||||
|
const key = this.#chunks[c]?.[i]
|
||||||
|
if (key === undefined || c < 0) return
|
||||||
|
if (reverse ? gte !== undefined && key < gte : lt !== undefined && key >= lt) return
|
||||||
|
yield key
|
||||||
|
if (this.#version !== version) {
|
||||||
|
version = this.#version
|
||||||
|
;[c, i] = this.#seek(key, !reverse)
|
||||||
|
if (reverse) [c, i] = this.#prev(c, i)
|
||||||
|
else [c, i] = this.#normalize(c, i)
|
||||||
|
} else if (reverse) [c, i] = this.#prev(c, i)
|
||||||
|
else [c, i] = this.#normalize(c, i + 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#insert(key: string): void {
|
||||||
|
const c = this.#chunkFor(key)
|
||||||
|
const chunk = this.#chunks[c]!
|
||||||
|
chunk.splice(lowerBound(chunk, key), 0, key)
|
||||||
|
if (chunk.length > this.maxChunk) this.#chunks.splice(c + 1, 0, chunk.splice(this.maxChunk >>> 1))
|
||||||
|
this.#version++
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Index of the last chunk whose first key is <= key (0 if none). */
|
||||||
|
#chunkFor(key: string): number {
|
||||||
|
const chunks = this.#chunks
|
||||||
let lo = 0
|
let lo = 0
|
||||||
let hi = keys.length
|
let hi = chunks.length - 1
|
||||||
|
while (lo < hi) {
|
||||||
|
const mid = (lo + hi + 1) >>> 1
|
||||||
|
if (chunks[mid]![0]! <= key) lo = mid
|
||||||
|
else hi = mid - 1
|
||||||
|
}
|
||||||
|
return lo
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Position of the first key `>= key` (`> key` when exclusive); may point one past a chunk's end. */
|
||||||
|
#seek(key: string, exclusive: boolean): [number, number] {
|
||||||
|
const c = this.#chunkFor(key)
|
||||||
|
const chunk = this.#chunks[c]!
|
||||||
|
let i = lowerBound(chunk, key)
|
||||||
|
if (exclusive && chunk[i] === key) i++
|
||||||
|
return [c, i]
|
||||||
|
}
|
||||||
|
|
||||||
|
#end(): [number, number] {
|
||||||
|
const c = this.#chunks.length - 1
|
||||||
|
return [c, this.#chunks[c]!.length]
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Moves a position that is past its chunk's end to the start of the next chunk. */
|
||||||
|
#normalize(c: number, i: number): [number, number] {
|
||||||
|
while (c < this.#chunks.length && i >= this.#chunks[c]!.length) {
|
||||||
|
c++
|
||||||
|
i = 0
|
||||||
|
}
|
||||||
|
return [c, i]
|
||||||
|
}
|
||||||
|
|
||||||
|
#prev(c: number, i: number): [number, number] {
|
||||||
|
i--
|
||||||
|
while (i < 0 && c > 0) {
|
||||||
|
c--
|
||||||
|
i = this.#chunks[c]!.length - 1
|
||||||
|
}
|
||||||
|
return i < 0 ? [-1, -1] : [c, i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function lowerBound(arr: string[], key: string): number {
|
||||||
|
let lo = 0
|
||||||
|
let hi = arr.length
|
||||||
while (lo < hi) {
|
while (lo < hi) {
|
||||||
const mid = (lo + hi) >>> 1
|
const mid = (lo + hi) >>> 1
|
||||||
const k = keys[mid]!
|
if (arr[mid]! < key) lo = mid + 1
|
||||||
if (k < key || (exclusive && k === key)) lo = mid + 1
|
|
||||||
else hi = mid
|
else hi = mid
|
||||||
}
|
}
|
||||||
return lo
|
return lo
|
||||||
}
|
}
|
||||||
|
|
||||||
keyAt(index: number): string | undefined {
|
|
||||||
return this.#keys[index]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function cmp(a: string, b: string): number {
|
function cmp(a: string, b: string): number {
|
||||||
return a < b ? -1 : a > b ? 1 : 0
|
return a < b ? -1 : a > b ? 1 : 0
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
import { describe, expect, it } from 'vitest'
|
||||||
|
import fc from 'fast-check'
|
||||||
|
import { ChunkedSortedMap } from '../../src/kv/sorted.js'
|
||||||
|
|
||||||
|
const key = fc.string({ unit: fc.constantFrom('a', 'b', 'c', '/', '0'), minLength: 1, maxLength: 4 })
|
||||||
|
type Op = { t: 'set'; k: string } | { t: 'del'; k: string } | { t: 'bulk'; ks: string[] }
|
||||||
|
const op: fc.Arbitrary<Op> = fc.oneof(
|
||||||
|
{ weight: 5, arbitrary: fc.record({ t: fc.constant('set' as const), k: key }) },
|
||||||
|
{ weight: 3, arbitrary: fc.record({ t: fc.constant('del' as const), k: key }) },
|
||||||
|
{ weight: 1, arbitrary: fc.record({ t: fc.constant('bulk' as const), ks: fc.array(key, { minLength: 60, maxLength: 120 }) }) },
|
||||||
|
)
|
||||||
|
|
||||||
|
describe('ChunkedSortedMap (model-based, tiny chunks)', () => {
|
||||||
|
it('matches a sorted reference for ranges in both directions', () => {
|
||||||
|
fc.assert(
|
||||||
|
fc.property(fc.array(op, { maxLength: 80 }), fc.integer({ min: 2, max: 6 }), key, key, (ops, chunk, a, b) => {
|
||||||
|
const m = new ChunkedSortedMap<number>(chunk)
|
||||||
|
const ref = new Map<string, number>()
|
||||||
|
ops.forEach((o, i) => {
|
||||||
|
if (o.t === 'set') (m.set(o.k, i), ref.set(o.k, i))
|
||||||
|
else if (o.t === 'del') expect(m.delete(o.k)).toBe(ref.delete(o.k))
|
||||||
|
else (m.setMany(o.ks.map((k) => [k, i])), o.ks.forEach((k) => ref.set(k, i)))
|
||||||
|
})
|
||||||
|
const sorted = [...ref.keys()].sort()
|
||||||
|
const [gte, lt] = a < b ? [a, b] : [b, a]
|
||||||
|
expect(m.size).toBe(ref.size)
|
||||||
|
expect([...m.keys({})]).toEqual(sorted)
|
||||||
|
expect([...m.keys({ reverse: true })]).toEqual([...sorted].reverse())
|
||||||
|
expect([...m.keys({ gte, lt })]).toEqual(sorted.filter((k) => k >= gte && k < lt))
|
||||||
|
expect([...m.keys({ gte, lt, reverse: true })]).toEqual(sorted.filter((k) => k >= gte && k < lt).reverse())
|
||||||
|
for (const k of sorted) expect(m.get(k)).toBe(ref.get(k))
|
||||||
|
}),
|
||||||
|
{ numRuns: 500 },
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('iterators resume correctly after mutations between steps', () => {
|
||||||
|
fc.assert(
|
||||||
|
fc.property(fc.uniqueArray(key, { minLength: 1, maxLength: 60 }), fc.array(fc.tuple(fc.boolean(), key), { maxLength: 40 }), fc.boolean(), (initial, muts, reverse) => {
|
||||||
|
const m = new ChunkedSortedMap<number>(3)
|
||||||
|
initial.forEach((k) => m.set(k, 0))
|
||||||
|
const live = new Set(initial)
|
||||||
|
const seen: string[] = []
|
||||||
|
let step = 0
|
||||||
|
for (const k of m.keys({ reverse })) {
|
||||||
|
seen.push(k)
|
||||||
|
const mut = muts[step++]
|
||||||
|
if (mut) {
|
||||||
|
const [add, mk] = mut
|
||||||
|
if (add) (m.set(mk, 1), live.add(mk))
|
||||||
|
else (m.delete(mk), live.delete(mk))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Strictly monotone, and every key present throughout the iteration was visited.
|
||||||
|
for (let i = 1; i < seen.length; i++) expect(reverse ? seen[i]! < seen[i - 1]! : seen[i]! > seen[i - 1]!).toBe(true)
|
||||||
|
const stable = initial.filter((k) => !muts.some(([, mk]) => mk === k))
|
||||||
|
for (const k of stable) expect(seen).toContain(k)
|
||||||
|
}),
|
||||||
|
{ numRuns: 500 },
|
||||||
|
)
|
||||||
|
})
|
||||||
|
})
|
||||||
Reference in New Issue
Block a user