Harvmaster/autocomplete-model
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Restructure into a monorepo: models, VS Code extension, daemon, shared

Browse Source 
- models/: GGUFs (Git LFS), manifest.json, model card, Apache-2.0 LICENSE + NOTICE
- extensions/vscode/: the Local Autocomplete extension (v0.5.0, publisher harvmaster)
- daemon/: acd completion daemon (bundled into the extension; also runs standalone)
- shared/: prompt layout + symbol extraction shared with the training pipeline
- scripts/: make-model-release.mjs, publish-extension-release.mjs (Gitea releases)
- README with quick start, settings, architecture, build and release instructions;
  MIT license for code; CHANGELOG

The default model manifest URL moves to models/manifest.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Harvmaster
2026-10-03 15:37:19 +00:00
co-authored by Claude Opus 5.5
parent f8dcf1b7df
commit fa8f775779
52 changed files with 309630 additions and 207 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+181
View File
@@ -0,0 +1,181 @@
# Editor integration design (VS Code + Zed)
Status (2026-10-03): the daemon, VS Code extension (incl. managed mode), symbols mode, `this.` repair and the edit recorder are implemented. See the [README](../README.md). Not yet built: token healing, the Zed shim, next-edit prediction.
## Goals
1. **Low latency while typing.** Ghost text within ~100 ms of a keystroke on the slowest machine (Ryzen laptop CPU), once warm. A slow cold start, when moving to a new location or loading new context, is acceptable.
2. **The same prompts as training.** The model only performs as measured if inference prompts are byte-identical in structure to training: suffix-first (SPM) order, repo-level header, the same symbol-context format.
3. **A context-mode toggle**: `symbols` (language-service context) vs `simple` (repo + path header only) vs `raw` (no header, for older models).
4. **One implementation for both editors.** All three machines run the same stack: laptop CPU, M3 Air and M1 Max.
5. **Private by default.** Everything runs locally. Telemetry is written to local files only, opt-in, with path exclusions.
## Architecture
```
┌──────────────┐ native API (HTTP, streaming) ┌──────────────────────────────────────┐
│ VS Code ext. │ ───────────────────────────────────▶ │ acd: local completion daemon (Node) │
└──────────────┘ + buffer sync, feedback events │ • prompt builder (pipeline/src/fim) │
│ • context modes + symbol provider │
┌──────────────┐ OpenAI-compatible /v1/completions │ (TS LanguageService, as training) │──▶ llama-server
│ Zed (built-in│ ───────────────────────────────────▶ │ • KV-cache-aware prompt layout │ (Metal / CPU)
│ provider) │ (prompt rewritten by acd) │ • cancellation, post-processing │
└──────────────┘ │ • local telemetry log │
└──────────────────────────────────────┘
```
**Why a daemon instead of logic in each extension:**
- **Training parity.** It imports `pipeline/src/fim.ts` (`fimPrompt`, `repoHeader`) and the extraction code behind `symbol-context.ts` directly, so production prompts can't drift from training prompts.
- **Same context everywhere.** Symbol context comes from the same TypeScript LanguageService code used to build the training data. VS Code's own hover/completion providers return differently formatted markdown, which the model has never seen.
- **Zed can't run our logic.** It has no extension hook for edit predictions; its built-in `open_ai_compatible_api` provider only calls an HTTP endpoint. A daemon serves both editors.
- **One server, one cache.** A single llama-server and KV cache per machine. The daemon can also run on the M1 Max and serve the laptop over the LAN.
## Prompt layout and KV cache
Training format (Stage C), in token order:
```
<|repo_name|>{repo}\n
[<|file_sep|>{dir}/__symbols__.d.ts\n{symbols}\n] ← only in `symbols` mode
<|file_sep|>{path}\n
<|fim_prefix|><|fim_suffix|>{suffix}<|fim_middle|>{prefix} → model generates middle + <|endoftext|>
```
Everything before the prefix is stable while typing, so each keystroke only appends to the end of the prompt. Measured on the laptop CPU: 3 tokens processed per keystroke, ~15 ms. To keep the prompt stable:
| Part | Changes when | Policy |
|---|---|---|
| Header + symbols | cursor moves to another line/scope, or a member-access receiver changes | Recompute off the hot path. Keep using the previous context until the new prompt is prefilled (stale-while-revalidate), so typing never waits on a context switch. |
| Suffix | text after the cursor changes (rare while typing) | First ≤ 512 tokens, cut at a line boundary. |
| Prefix | every keystroke | **Chunked truncation:** drop from the start in 256-token blocks aligned to line starts, so the window start only moves at block boundaries. Truncating to "the last N tokens" on every keystroke would shift the start and invalidate the whole cache. Training used variable prefix lengths, so any line-aligned start is in distribution. |
llama-server settings: `cache_prompt: true`, two slots (one for live requests, one for background warming of a new context), pinned per editor window with `id_slot`.
## Context modes (the toggle)
| Mode | Prompt | Cost | Use |
|---|---|---|---|
| `symbols` | header + `__symbols__.d.ts` (budget configurable, default 2,048 tokens) + path | context extraction ~40 ms (measured), computed off the hot path; prefill of the larger header on context changes | default for TS/JS/Vue once Stage C is validated |
| `simple` | header with repo + path only | none | fallback; non-TS languages; language service still loading. Stage C trained 20% of samples this way (context dropout), so it is in distribution |
| `raw` | no header | none | Stage A/B models, and A/B comparisons |
**Model profiles** (`acd.config.json`) declare which modes and orders a model supports, e.g. `stage-c: { order: spm, modes: [symbols, simple] }`, `stage-b: { order: spm, modes: [raw] }`, `base-0.5b: { order: psm, modes: [raw] }`. The toggle only offers valid modes.
**Symbol provider (in acd):**
- One TS LanguageService per workspace root, with tsconfig discovery.
- Vue SFCs go through the same `regionsOf` script extraction as training.
- Unsaved buffers come from the editor (VS Code: `/v1/documents/sync`; Zed: the request's own prefix + suffix merged over the on-disk file).
- Output format and ranking are identical to `symbol-context.ts`: member completions at `expr.`, then imported symbols used near the cursor, then other imports. Signatures only; framework modules (`vue`, `pinia`, `node:*`) skipped.
- Cached per anchor (file, line/scope, receiver).
## Daemon API (acd)
| Endpoint | Purpose |
|---|---|
| `POST /v1/fim` | Native completion request: `{ requestId, workspaceRoot, path, languageId, documentVersion, prefix, suffix, contextMode?, maxLines?, singleLine? }`. Streams SSE `{ text, done, timings, meanLogprob }`. Closing the connection cancels the llama request. |
| `POST /v1/completions` | OpenAI-compatible shim for Zed (below). |
| `POST /v1/documents/sync` | Unsaved buffer contents for the language service (debounced). |
| `POST /v1/feedback` | `{ requestId, event: shown | accepted | partial | rejected, acceptedText?, survivedAfter30s? }`, appended to the local telemetry log. |
| `GET/PUT /v1/config` | Current model and context mode; lets the toggle be changed from either editor or a CLI (`acctl mode symbols`). |
| `GET /health` | Readiness: model loaded, language-service status per root. |
**Token healing (when the cursor is mid-identifier):** the prefix ends partway through a word (`this.connections.del▮`), so its last token is an unnatural split. In suffix-first order the model then tends to treat the word as finished: it emits `(relayUrl)` instead of `ete(relayUrl)`, consistently across Stage B and C in the scenario test, and about 3 points worse than prefix-first on the 99 mid-identifier eval samples. Fix:
1. Drop the prefix's last token.
2. Constrain generation (llama-server `grammar`, or logit bias) to start with that token's text.
3. Strip it again before returning.
Measure with the eval harness's mid-identifier subset.
**Post-processing**, shared with `eval.ts` so evaluation measures what users see:
- Trim text that repeats the existing suffix.
- Enforce single-line mode when the cursor has non-whitespace text after it on the same line.
- Cap multi-line completions (default 8 lines).
- Fix indentation to the current line.
- Drop empty or whitespace-only results.
- Optionally suppress completions below a mean-logprob threshold, calibrated later from telemetry.
## VS Code extension
- `vscode.languages.registerInlineCompletionItemProvider` for `typescript`, `javascript`, `typescriptreact`, `vue`, `json`, `jsonc`, `yaml`, `dockercompose`, `dockerfile`.
- **Request flow:** in `provideInlineCompletionItems`:
1. Apply a ~25 ms debounce.
2. Send prefix, suffix, path and version to `/v1/fim`.
3. Tie `token.onCancellationRequested` to aborting the fetch.
4. Return one `InlineCompletionItem`, with a `command` that reports `accepted`.
- **Suggest widget interplay:** when `context.selectedCompletionInfo` is set (the IntelliSense list is open), send the selected item's text as part of the prefix so the ghost text continues it. Otherwise VS Code discards the suggestion.
- **Partial accept** (word/line) uses VS Code's built-in commands. Reporting partial accepts and "shown" events needs the proposed `inlineCompletionsAdditions` API (`enabledApiProposals`). That's fine for a privately installed VSIX, but such an extension can't be published to the Marketplace. The fallback is acceptance heuristics in the daemon, see Telemetry.
- **Status bar:** `AC: symbols · 42 ms`. Clicking it opens a quick pick for the context mode (and model, if several are configured). Command: `autocomplete.toggleContextMode`. Setting: `autocomplete.contextMode` (`symbols | simple | raw`), plus `autocomplete.server` (URL; default spawns a local acd).
- **Privacy controls:** `autocomplete.excludeGlobs` (never send these files, defaulting to `.env*`, `*.pem`, …), `autocomplete.telemetry` (off by default), `autocomplete.recordEdits` (off by default; the next-edit data recorder).
## Zed
Zed (2026) has built-in edit-prediction providers, including `open_ai_compatible_api`. Third-party extensions can't provide edit predictions ([discussion #30855](https://github.com/zed-industries/zed/discussions/30855)), so **Zed needs no extension, only settings that point at acd**:
```jsonc
{
"edit_predictions": {
"provider": "open_ai_compatible_api",
"open_ai_compatible_api": {
"api_url": "http://127.0.0.1:7071/v1/completions",
"model": "ac:symbols", // the toggle: "ac:symbols" | "ac:simple"
"prompt_format": "qwen"
}
}
}
```
**How acd handles a Zed request:**
1. Zed builds its own Qwen-style FIM prompt and sends it to `/v1/completions`.
2. acd parses `prefix` and `suffix` out of Zed's FIM tokens.
3. It resolves the file. If the prompt contains a `<|file_sep|>path`, it uses that; otherwise it locates the buffer by matching the prefix against files under configured workspace roots.
4. It **rebuilds the prompt in our training layout** (SPM + header + symbols) and streams the result back in OpenAI completions format.
5. The context mode comes from the `model` string, so the toggle is a one-word settings edit. `acctl mode` works too.
**Spike to do first (unverified):** capture real Zed requests with a logging proxy, to confirm:
- whether the file path is included
- the prefix/suffix truncation sizes
- streaming, `max_tokens` and stop sequences
- whether Zed aborts superseded requests, so acd must cancel them
- that the `model` string is passed through verbatim
If Zed's prefix/suffix are heavily truncated, acd merges them over the on-disk file to recover full context.
**Telemetry in Zed:** Zed doesn't report accept/reject to providers. acd infers acceptance: if the next request's prefix ends with the previously predicted text, it was accepted (the same heuristic serves as the fallback for VS Code).
## Runtimes per machine
| Machine | llama-server backend | Notes |
|---|---|---|
| Ryzen 7945HX laptop | CPU (AVX-512), 16 threads | Measured: 1.7k tok/s prefill, ~95 tok/s decode for 0.5B Q8_0. Try the Vulkan iGPU build as an experiment. |
| M3 Air | Metal | Fanless: measure sustained throughput, not just bursts. |
| M1 Max | Metal | Fastest. Can also serve the other machines over the LAN (`autocomplete.server`). |
Model: `stage-c-q8_0.gguf` (531 MB) once validated; `stage-b` as the `raw`-mode fallback. Try Q6_K / Q4_K_M only if the M3 Air's latency needs it, and re-run the eval if so.
## Telemetry (local only, opt-in)
JSONL under the daemon's data directory, one record per request:
- timings: debounce, context, prefill, first token, total
- mode, model, prediction (hashed or plaintext per setting)
- outcome: shown, accepted, partial, rejected
- **survival**: does the accepted text still exist 30 s later?
That's what's needed to calibrate suppression and to compare `symbols` vs `simple` in real use. The **edit recorder** (separate opt-in) logs document diffs with timestamps, for future next-edit training data.
## Build order
1. **acd core:** `/v1/fim` with `simple`/`raw`, chunked prefix truncation, llama-server management, cancellation, shared post-processing. Validate with `bench-typing.ts` pointed at acd: warm keystroke latency on all three machines.
2. **VS Code extension:** inline provider, status-bar toggle, settings, exclusions.
3. **`symbols` mode:** move the extractor from `symbol-context.ts` into a shared module used by acd and the training pipeline, add workspace TS services, buffer sync and stale-while-revalidate warming.
4. **Zed:** the logging-proxy spike, then the `/v1/completions` shim.
5. **Telemetry + suppression threshold**, then the edit recorder.
## Proposed repo layout
```
shared/ prompt builder, post-processing, symbol extraction (also used by the training pipeline)
daemon/ acd (Node/TS): HTTP API, llama-server manager, workspace LanguageServices, telemetry
extensions/vscode/ VS Code extension (bundles the daemon; managed mode downloads model + llama.cpp)
extensions/zed/ (planned) settings snippet + README
models/ GGUF models (Git LFS) + manifest.json
```