autocomplete-model/docs/architecture.md

# Editor integration design (VS Code + Zed)

Status (2026-10-03): the daemon, VS Code extension (incl. managed mode), symbols mode (format 1 and 2), TS errors context, `this.` repair, token healing, next-edit plumbing (`/v1/next-edit`, decorations UI, shared prompt format in `shared/nextedit.ts`), pre-warming, the debug log and the edit recorder are implemented. See the [README](../README.md). Not yet built: the Zed shim, parallel "as if accepted" completions, recent-edits-in-other-files context.

## Goals

1. **Low latency while typing.** Ghost text within ~100 ms of a keystroke on the slowest machine (Ryzen laptop CPU), once warm. A slow cold start, when moving to a new location or loading new context, is acceptable.
2. **The same prompts as training.** The model only performs as measured if inference prompts are byte-identical in structure to training: suffix-first (SPM) order, repo-level header, the same symbol-context format.
3. **A context-mode toggle**: `symbols` (language-service context) vs `simple` (repo + path header only) vs `raw` (no header, for older models).
4. **One implementation for both editors.** All three machines run the same stack: laptop CPU, M3 Air and M1 Max.
5. **Private by default.** Everything runs locally. Telemetry is written to local files only, opt-in, with path exclusions.

## Architecture

```
┌──────────────┐   native API (HTTP, streaming)       ┌──────────────────────────────────────┐
│ VS Code ext. │ ───────────────────────────────────▶ │  acd: local completion daemon (Node) │
└──────────────┘   + buffer sync, feedback events     │  • prompt builder (pipeline/src/fim) │
                                                      │  • context modes + symbol provider   │
┌──────────────┐   OpenAI-compatible /v1/completions  │    (TS LanguageService, as training) │──▶ llama-server
│ Zed (built-in│ ───────────────────────────────────▶ │  • KV-cache-aware prompt layout      │    (Metal / CPU)
│  provider)   │   (prompt rewritten by acd)          │  • cancellation, post-processing     │
└──────────────┘                                      │  • local telemetry log               │
                                                      └──────────────────────────────────────┘
```

**Why a daemon instead of logic in each extension:**
- **Training parity.** It imports `pipeline/src/fim.ts` (`fimPrompt`, `repoHeader`) and the extraction code behind `symbol-context.ts` directly, so production prompts can't drift from training prompts.
- **Same context everywhere.** Symbol context comes from the same TypeScript LanguageService code used to build the training data. VS Code's own hover/completion providers return differently formatted markdown, which the model has never seen.
- **Zed can't run our logic.** It has no extension hook for edit predictions; its built-in `open_ai_compatible_api` provider only calls an HTTP endpoint. A daemon serves both editors.
- **One server, one cache.** A single llama-server and KV cache per machine. The daemon can also run on the M1 Max and serve the laptop over the LAN.

## Prompt layout and KV cache

Training format (Stage C), in token order:

```
<|repo_name|>{repo}\n
[<|file_sep|>{dir}/__symbols__.d.ts\n{symbols}\n]      ← only in `symbols` mode
<|file_sep|>{path}\n
<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix}   → model generates middle + <|endoftext|>
```

Everything before the prefix is stable while typing, so each keystroke only appends to the end of the prompt. Measured on the laptop CPU: 3 tokens processed per keystroke, ~15 ms. To keep the prompt stable:

| Part | Changes when | Policy |
|---|---|---|
| Header + symbols | cursor moves to another line/scope, or a member-access receiver changes | Recompute off the hot path. Keep using the previous context until the new prompt is prefilled (stale-while-revalidate), so typing never waits on a context switch. |
| Suffix | text after the cursor changes (rare while typing) | First ≤ 512 tokens, cut at a line boundary. |
| Prefix | every keystroke | **Chunked truncation:** drop from the start in 256-token blocks aligned to line starts, so the window start only moves at block boundaries. Truncating to "the last N tokens" on every keystroke would shift the start and invalidate the whole cache. Training used variable prefix lengths, so any line-aligned start is in distribution. |

llama-server settings: `cache_prompt: true`, two slots (one for live requests, one for background warming of a new context), pinned per editor window with `id_slot`.

## Context modes (the toggle)

| Mode | Prompt | Cost | Use |
|---|---|---|---|
| `symbols` | header + `__symbols__.d.ts` (budget configurable, default 2,048 tokens) + path | context extraction ~40 ms (measured), computed off the hot path; prefill of the larger header on context changes | default for TS/JS/Vue once Stage C is validated |
| `simple` | header with repo + path only | none | fallback; non-TS languages; language service still loading. Stage C trained 20% of samples this way (context dropout), so it is in distribution |
| `raw` | no header | none | Stage A/B models, and A/B comparisons |

**Model profiles** (`acd.config.json`) declare which modes and orders a model supports, e.g. `stage-c: { order: spm, modes: [symbols, simple] }`, `stage-b: { order: spm, modes: [raw] }`, `base-0.5b: { order: psm, modes: [raw] }`. The toggle only offers valid modes.

**Symbol provider (in acd):**
- One TS LanguageService per workspace root, with tsconfig discovery.
- Vue SFCs go through the same `regionsOf` script extraction as training.
- Unsaved buffers come from the editor (VS Code: `/v1/documents/sync`; Zed: the request's own prefix + suffix merged over the on-disk file).
- Output format and ranking are identical to `symbol-context.ts`: member completions at `expr.`, then imported symbols used near the cursor, then other imports. Signatures only; framework modules (`vue`, `pinia`, `node:*`) skipped.
- Cached per anchor (file, line/scope, receiver).

## Daemon API (acd)

| Endpoint | Purpose |
|---|---|
| `POST /v1/fim` | Native completion request: `{ requestId, workspaceRoot, path, languageId, documentVersion, prefix, suffix, contextMode?, maxLines?, singleLine? }`. Streams SSE `{ text, done, timings, meanLogprob }`. Closing the connection cancels the llama request. |
| `POST /v1/completions` | OpenAI-compatible shim for Zed (below). |
| `POST /v1/documents/sync` | Unsaved buffer contents for the language service (debounced). |
| `POST /v1/feedback` | `{ requestId, event: shown | accepted | partial | rejected, acceptedText?, survivedAfter30s? }`, appended to the local telemetry log. |
| `GET/PUT /v1/config` | Current model and context mode; lets the toggle be changed from either editor or a CLI (`acctl mode symbols`). |
| `GET /health` | Readiness: model loaded, language-service status per root. |

**Token healing (when the cursor is mid-identifier):** the prefix ends partway through a word (`this.connections.del▮`), so its last token is an unnatural split. In suffix-first order the model then tends to treat the word as finished: it emits `(relayUrl)` instead of `ete(relayUrl)`, consistently across Stage B and C in the scenario test, and about 3 points worse than prefix-first on the 99 mid-identifier eval samples. Fix:
1. Drop the prefix's last token.
2. Constrain generation (llama-server `grammar`, or logit bias) to start with that token's text.
3. Strip it again before returning.

Measure with the eval harness's mid-identifier subset.

**Post-processing**, shared with `eval.ts` so evaluation measures what users see:
- Trim text that repeats the existing suffix.
- Enforce single-line mode when the cursor has non-whitespace text after it on the same line.
- Cap multi-line completions (default 8 lines).
- Fix indentation to the current line.
- Drop empty or whitespace-only results.
- Optionally suppress completions below a mean-logprob threshold, calibrated later from telemetry.

## VS Code extension

- `vscode.languages.registerInlineCompletionItemProvider` for `typescript`, `javascript`, `typescriptreact`, `vue`, `json`, `jsonc`, `yaml`, `dockercompose`, `dockerfile`.
- **Request flow:** in `provideInlineCompletionItems`:
  1. Apply a ~25 ms debounce.
  2. Send prefix, suffix, path and version to `/v1/fim`.
  3. Tie `token.onCancellationRequested` to aborting the fetch.
  4. Return one `InlineCompletionItem`, with a `command` that reports `accepted`.
- **Suggest widget interplay:** when `context.selectedCompletionInfo` is set (the IntelliSense list is open), send the selected item's text as part of the prefix so the ghost text continues it. Otherwise VS Code discards the suggestion.
- **Partial accept** (word/line) uses VS Code's built-in commands. Reporting partial accepts and "shown" events needs the proposed `inlineCompletionsAdditions` API (`enabledApiProposals`). That's fine for a privately installed VSIX, but such an extension can't be published to the Marketplace. The fallback is acceptance heuristics in the daemon, see Telemetry.
- **Status bar:** `AC: symbols · 42 ms`. Clicking it opens a quick pick for the context mode (and model, if several are configured). Command: `autocomplete.toggleContextMode`. Setting: `autocomplete.contextMode` (`symbols | simple | raw`), plus `autocomplete.server` (URL; default spawns a local acd).
- **Privacy controls:** `autocomplete.excludeGlobs` (never send these files, defaulting to `.env*`, `*.pem`, …), `autocomplete.telemetry` (off by default), `autocomplete.recordEdits` (off by default; the next-edit data recorder).

## Zed

Zed (2026) has built-in edit-prediction providers, including `open_ai_compatible_api`. Third-party extensions can't provide edit predictions ([discussion #30855](https://github.com/zed-industries/zed/discussions/30855)), so **Zed needs no extension, only settings that point at acd**:

```jsonc
{
  "edit_predictions": {
    "provider": "open_ai_compatible_api",
    "open_ai_compatible_api": {
      "api_url": "http://127.0.0.1:7071/v1/completions",
      "model": "ac:symbols",          // the toggle: "ac:symbols" | "ac:simple"
      "prompt_format": "qwen"
    }
  }
}
```

**How acd handles a Zed request:**
1. Zed builds its own Qwen-style FIM prompt and sends it to `/v1/completions`.
2. acd parses `prefix` and `suffix` out of Zed's FIM tokens.
3. It resolves the file. If the prompt contains a `<|file_sep|>path`, it uses that; otherwise it locates the buffer by matching the prefix against files under configured workspace roots.
4. It **rebuilds the prompt in our training layout** (SPM + header + symbols) and streams the result back in OpenAI completions format.
5. The context mode comes from the `model` string, so the toggle is a one-word settings edit. `acctl mode` works too.

**Spike to do first (unverified):** capture real Zed requests with a logging proxy, to confirm:
- whether the file path is included
- the prefix/suffix truncation sizes
- streaming, `max_tokens` and stop sequences
- whether Zed aborts superseded requests, so acd must cancel them
- that the `model` string is passed through verbatim

If Zed's prefix/suffix are heavily truncated, acd merges them over the on-disk file to recover full context.

**Telemetry in Zed:** Zed doesn't report accept/reject to providers. acd infers acceptance: if the next request's prefix ends with the previously predicted text, it was accepted (the same heuristic serves as the fallback for VS Code).

## Runtimes per machine

| Machine | llama-server backend | Notes |
|---|---|---|
| Ryzen 7945HX laptop | CPU (AVX-512), 16 threads | Measured: 1.7k tok/s prefill, ~95 tok/s decode for 0.5B Q8_0. Try the Vulkan iGPU build as an experiment. |
| M3 Air | Metal | Fanless: measure sustained throughput, not just bursts. |
| M1 Max | Metal | Fastest. Can also serve the other machines over the LAN (`autocomplete.server`). |

Model: `stage-c-q8_0.gguf` (531 MB) once validated; `stage-b` as the `raw`-mode fallback. Try Q6_K / Q4_K_M only if the M3 Air's latency needs it, and re-run the eval if so.

## Telemetry (local only, opt-in)

JSONL under the daemon's data directory, one record per request:
- timings: debounce, context, prefill, first token, total
- mode, model, prediction (hashed or plaintext per setting)
- outcome: shown, accepted, partial, rejected
- **survival**: does the accepted text still exist 30 s later?

That's what's needed to calibrate suppression and to compare `symbols` vs `simple` in real use. The **edit recorder** (separate opt-in) logs document diffs with timestamps, for future next-edit training data.

## Build order

1. **acd core:** `/v1/fim` with `simple`/`raw`, chunked prefix truncation, llama-server management, cancellation, shared post-processing. Validate with `bench-typing.ts` pointed at acd: warm keystroke latency on all three machines.
2. **VS Code extension:** inline provider, status-bar toggle, settings, exclusions.
3. **`symbols` mode:** move the extractor from `symbol-context.ts` into a shared module used by acd and the training pipeline, add workspace TS services, buffer sync and stale-while-revalidate warming.
4. **Zed:** the logging-proxy spike, then the `/v1/completions` shim.
5. **Telemetry + suppression threshold**, then the edit recorder.

## Proposed repo layout

```
shared/             prompt builder, post-processing, symbol extraction (also used by the training pipeline)
daemon/             acd (Node/TS): HTTP API, llama-server manager, workspace LanguageServices, telemetry
extensions/vscode/  VS Code extension (bundles the daemon; managed mode downloads model + llama.cpp)
extensions/zed/     (planned) settings snippet + README
models/             GGUF models (Git LFS) + manifest.json
```