autocomplete-model/docs/architecture.md

Editor integration design (VS Code + Zed)

Status (2026-10-03): the daemon, VS Code extension (incl. managed mode), symbols mode, this. repair and the edit recorder are implemented. See the README. Not yet built: token healing, the Zed shim, next-edit prediction.

Goals

1. Low latency while typing. Ghost text within ~100 ms of a keystroke on the slowest machine (Ryzen laptop CPU), once warm. A slow cold start, when moving to a new location or loading new context, is acceptable.
2. The same prompts as training. The model only performs as measured if inference prompts are byte-identical in structure to training: suffix-first (SPM) order, repo-level header, the same symbol-context format.
3. A context-mode toggle: symbols (language-service context) vs simple (repo + path header only) vs raw (no header, for older models).
4. One implementation for both editors. All three machines run the same stack: laptop CPU, M3 Air and M1 Max.
5. Private by default. Everything runs locally. Telemetry is written to local files only, opt-in, with path exclusions.

Architecture

┌──────────────┐   native API (HTTP, streaming)       ┌──────────────────────────────────────┐
│ VS Code ext. │ ───────────────────────────────────▶ │  acd: local completion daemon (Node) │
└──────────────┘   + buffer sync, feedback events     │  • prompt builder (pipeline/src/fim) │
                                                      │  • context modes + symbol provider   │
┌──────────────┐   OpenAI-compatible /v1/completions  │    (TS LanguageService, as training) │──▶ llama-server
│ Zed (built-in│ ───────────────────────────────────▶ │  • KV-cache-aware prompt layout      │    (Metal / CPU)
│  provider)   │   (prompt rewritten by acd)          │  • cancellation, post-processing     │
└──────────────┘                                      │  • local telemetry log               │
                                                      └──────────────────────────────────────┘

Why a daemon instead of logic in each extension:

• Training parity. It imports pipeline/src/fim.ts (fimPrompt, repoHeader) and the extraction code behind symbol-context.ts directly, so production prompts can't drift from training prompts.
• Same context everywhere. Symbol context comes from the same TypeScript LanguageService code used to build the training data. VS Code's own hover/completion providers return differently formatted markdown, which the model has never seen.
• Zed can't run our logic. It has no extension hook for edit predictions; its built-in open_ai_compatible_api provider only calls an HTTP endpoint. A daemon serves both editors.
• One server, one cache. A single llama-server and KV cache per machine. The daemon can also run on the M1 Max and serve the laptop over the LAN.

Prompt layout and KV cache

Training format (Stage C), in token order:

<|repo_name|>{repo}\n
[<|file_sep|>{dir}/__symbols__.d.ts\n{symbols}\n]      ← only in `symbols` mode
<|file_sep|>{path}\n
<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix}   → model generates middle + <|endoftext|>

Everything before the prefix is stable while typing, so each keystroke only appends to the end of the prompt. Measured on the laptop CPU: 3 tokens processed per keystroke, ~15 ms. To keep the prompt stable:

Part	Changes when	Policy
Header + symbols	cursor moves to another line/scope, or a member-access receiver changes	Recompute off the hot path. Keep using the previous context until the new prompt is prefilled (stale-while-revalidate), so typing never waits on a context switch.
Suffix	text after the cursor changes (rare while typing)	First ≤ 512 tokens, cut at a line boundary.
Prefix	every keystroke	Chunked truncation: drop from the start in 256-token blocks aligned to line starts, so the window start only moves at block boundaries. Truncating to "the last N tokens" on every keystroke would shift the start and invalidate the whole cache. Training used variable prefix lengths, so any line-aligned start is in distribution.

llama-server settings: cache_prompt: true, two slots (one for live requests, one for background warming of a new context), pinned per editor window with id_slot.

Context modes (the toggle)

Mode	Prompt	Cost	Use
symbols	header + __symbols__.d.ts (budget configurable, default 2,048 tokens) + path	context extraction ~40 ms (measured), computed off the hot path; prefill of the larger header on context changes	default for TS/JS/Vue once Stage C is validated
simple	header with repo + path only	none	fallback; non-TS languages; language service still loading. Stage C trained 20% of samples this way (context dropout), so it is in distribution
raw	no header	none	Stage A/B models, and A/B comparisons

Model profiles (acd.config.json) declare which modes and orders a model supports, e.g. stage-c: { order: spm, modes: [symbols, simple] }, stage-b: { order: spm, modes: [raw] }, base-0.5b: { order: psm, modes: [raw] }. The toggle only offers valid modes.

Symbol provider (in acd):

• One TS LanguageService per workspace root, with tsconfig discovery.
• Vue SFCs go through the same regionsOf script extraction as training.
• Unsaved buffers come from the editor (VS Code: /v1/documents/sync; Zed: the request's own prefix + suffix merged over the on-disk file).
• Output format and ranking are identical to symbol-context.ts: member completions at expr., then imported symbols used near the cursor, then other imports. Signatures only; framework modules (vue, pinia, node:*) skipped.
• Cached per anchor (file, line/scope, receiver).

Daemon API (acd)

Endpoint	Purpose
POST /v1/fim	Native completion request: { requestId, workspaceRoot, path, languageId, documentVersion, prefix, suffix, contextMode?, maxLines?, singleLine? }. Streams SSE { text, done, timings, meanLogprob }. Closing the connection cancels the llama request.
POST /v1/completions	OpenAI-compatible shim for Zed (below).
POST /v1/documents/sync	Unsaved buffer contents for the language service (debounced).
POST /v1/feedback	`{ requestId, event: shown
GET/PUT /v1/config	Current model and context mode; lets the toggle be changed from either editor or a CLI (acctl mode symbols).
GET /health	Readiness: model loaded, language-service status per root.

Token healing (when the cursor is mid-identifier): the prefix ends partway through a word (this.connections.del▮), so its last token is an unnatural split. In suffix-first order the model then tends to treat the word as finished: it emits (relayUrl) instead of ete(relayUrl), consistently across Stage B and C in the scenario test, and about 3 points worse than prefix-first on the 99 mid-identifier eval samples. Fix:

1. Drop the prefix's last token.
2. Constrain generation (llama-server grammar, or logit bias) to start with that token's text.
3. Strip it again before returning.

Measure with the eval harness's mid-identifier subset.

Post-processing, shared with eval.ts so evaluation measures what users see:

• Trim text that repeats the existing suffix.
• Enforce single-line mode when the cursor has non-whitespace text after it on the same line.
• Cap multi-line completions (default 8 lines).
• Fix indentation to the current line.
• Drop empty or whitespace-only results.
• Optionally suppress completions below a mean-logprob threshold, calibrated later from telemetry.

VS Code extension

• vscode.languages.registerInlineCompletionItemProvider for typescript, javascript, typescriptreact, vue, json, jsonc, yaml, dockercompose, dockerfile.
• Request flow: in provideInlineCompletionItems:
  1. Apply a ~25 ms debounce.
  2. Send prefix, suffix, path and version to /v1/fim.
  3. Tie token.onCancellationRequested to aborting the fetch.
  4. Return one InlineCompletionItem, with a command that reports accepted.
• Suggest widget interplay: when context.selectedCompletionInfo is set (the IntelliSense list is open), send the selected item's text as part of the prefix so the ghost text continues it. Otherwise VS Code discards the suggestion.
• Partial accept (word/line) uses VS Code's built-in commands. Reporting partial accepts and "shown" events needs the proposed inlineCompletionsAdditions API (enabledApiProposals). That's fine for a privately installed VSIX, but such an extension can't be published to the Marketplace. The fallback is acceptance heuristics in the daemon, see Telemetry.
• Status bar: AC: symbols · 42 ms. Clicking it opens a quick pick for the context mode (and model, if several are configured). Command: autocomplete.toggleContextMode. Setting: autocomplete.contextMode (symbols | simple | raw), plus autocomplete.server (URL; default spawns a local acd).
• Privacy controls: autocomplete.excludeGlobs (never send these files, defaulting to .env*, *.pem, …), autocomplete.telemetry (off by default), autocomplete.recordEdits (off by default; the next-edit data recorder).

Zed

Zed (2026) has built-in edit-prediction providers, including open_ai_compatible_api. Third-party extensions can't provide edit predictions (discussion #30855), so Zed needs no extension, only settings that point at acd:

{
  "edit_predictions": {
    "provider": "open_ai_compatible_api",
    "open_ai_compatible_api": {
      "api_url": "http://127.0.0.1:7071/v1/completions",
      "model": "ac:symbols",          // the toggle: "ac:symbols" | "ac:simple"
      "prompt_format": "qwen"
    }
  }
}

How acd handles a Zed request:

1. Zed builds its own Qwen-style FIM prompt and sends it to /v1/completions.
2. acd parses prefix and suffix out of Zed's FIM tokens.
3. It resolves the file. If the prompt contains a <|file_sep|>path, it uses that; otherwise it locates the buffer by matching the prefix against files under configured workspace roots.
4. It rebuilds the prompt in our training layout (SPM + header + symbols) and streams the result back in OpenAI completions format.
5. The context mode comes from the model string, so the toggle is a one-word settings edit. acctl mode works too.

Spike to do first (unverified): capture real Zed requests with a logging proxy, to confirm:

• whether the file path is included
• the prefix/suffix truncation sizes
• streaming, max_tokens and stop sequences
• whether Zed aborts superseded requests, so acd must cancel them
• that the model string is passed through verbatim

If Zed's prefix/suffix are heavily truncated, acd merges them over the on-disk file to recover full context.

Telemetry in Zed: Zed doesn't report accept/reject to providers. acd infers acceptance: if the next request's prefix ends with the previously predicted text, it was accepted (the same heuristic serves as the fallback for VS Code).

Runtimes per machine

Machine	llama-server backend	Notes
Ryzen 7945HX laptop	CPU (AVX-512), 16 threads	Measured: 1.7k tok/s prefill, ~95 tok/s decode for 0.5B Q8_0. Try the Vulkan iGPU build as an experiment.
M3 Air	Metal	Fanless: measure sustained throughput, not just bursts.
M1 Max	Metal	Fastest. Can also serve the other machines over the LAN (autocomplete.server).

Model: stage-c-q8_0.gguf (531 MB) once validated; stage-b as the raw-mode fallback. Try Q6_K / Q4_K_M only if the M3 Air's latency needs it, and re-run the eval if so.

Telemetry (local only, opt-in)

JSONL under the daemon's data directory, one record per request:

• timings: debounce, context, prefill, first token, total
• mode, model, prediction (hashed or plaintext per setting)
• outcome: shown, accepted, partial, rejected
• survival: does the accepted text still exist 30 s later?

That's what's needed to calibrate suppression and to compare symbols vs simple in real use. The edit recorder (separate opt-in) logs document diffs with timestamps, for future next-edit training data.

Build order

1. acd core: /v1/fim with simple/raw, chunked prefix truncation, llama-server management, cancellation, shared post-processing. Validate with bench-typing.ts pointed at acd: warm keystroke latency on all three machines.
2. VS Code extension: inline provider, status-bar toggle, settings, exclusions.
3. symbols mode: move the extractor from symbol-context.ts into a shared module used by acd and the training pipeline, add workspace TS services, buffer sync and stale-while-revalidate warming.
4. Zed: the logging-proxy spike, then the /v1/completions shim.
5. Telemetry + suppression threshold, then the edit recorder.

Proposed repo layout

shared/             prompt builder, post-processing, symbol extraction (also used by the training pipeline)
daemon/             acd (Node/TS): HTTP API, llama-server manager, workspace LanguageServices, telemetry
extensions/vscode/  VS Code extension (bundles the daemon; managed mode downloads model + llama.cpp)
extensions/zed/     (planned) settings snippet + README
models/             GGUF models (Git LFS) + manifest.json