Before the search: what Index workspace reads, skips, hashes and keeps when you cancel

Press Index workspace in twinny’s Embeddings tab and a progress bar walks through your files. An earlier post covered what happens after that, when a question is matched against the index: the two searches, the reranker, the labelled blocks in the prompt. It also said what a chunk is, so none of that is repeated here. This post is about the run itself: what gets read, why the second run is fast, what the embedding server sees, and what the odd cases do. The code is src/extension/embeddings/indexer.ts, embedder.ts and indexable.ts.

What a scan reads

The walk is an allowlist. A file is considered only if its extension or its whole name is on a list in indexable.ts: about 150 extensions covering source, config, data formats and documentation, plus names like Makefile, Dockerfile, README and .gitignore. Anything not on the list is never opened. Lockfiles (package-lock.json, Cargo.lock, go.sum and so on) and files ending .min.js, .min.css, .map, .snap, .lock or .svg are skipped by name, SVG because it is text that is never an answer to a question. A wrapped name such as config.yaml.example or README.md.bak is judged by the inner extension when the outer one is bak, example, sample, template, dist or orig.

On top of that, each workspace folder’s root .gitignore is applied, with twinny.embeddingIgnoredGlobs added to it, and .git and git submodules are always left out. Empty files and files over 512 KB are dropped at stat time, before anything is read. The last check runs after the read: a NUL byte in the first 8,000 bytes marks the file as binary, the same heuristic git uses, which catches a gzipped file wearing a text extension.

Two ways to be unchanged

A scan produces a size and a modification time for every file. The index keeps a manifest, manifest.json next to the LanceDB table under ~/.twinny/embeddings/, with a stamp per file: size, mtime, a SHA-1 of the contents and the number of chunks.

Update compares the two in order. If size and mtime match the stamp, the file is not even opened. If they differ, the file is read and hashed, and if the hash matches the stamp, it is still unchanged: the stamp gets the new mtime and no request is made. So a touch across the tree, or a formatter that rewrites a file byte for byte, costs a read per file and no embedding. Only a file whose contents actually differ goes to the server, and a file in the manifest that the scan no longer finds is deleted from the table.

That is why the second run reports Index is up to date with nothing embedded, while the first run took minutes. The manifest has a version number too; one from an older layout is treated as empty, and the next run embeds everything again.

What the server sees

Changed files are embedded through a queue, four files in flight at once against a local server. Against a team gateway it is one at a time, so an index run does not crowd a server the whole team shares.

Each chunk is sent with its path in front, in windows of 600 characters when it is longer than that. The resulting texts go to the server 16 per request, and the difference between a list per request and one text per request is, as the comment in embedder.ts puts it, the difference between minutes and most of an hour. Not every server accepts a list. If the first list request fails with a 4xx, the embedder takes that as “one string at a time” and sends singles for the rest of the session. A connection failure, a 5xx or a busy refusal is not treated that way; those are real and stop the run with the server’s message.

A busy server gets patience first. On a 429 the request is retried after 500 ms, then 1, 2, 4, 8, 8 and 8 seconds. Once those are spent the refusal stands. Each request has a 60-second timeout.

Rows are written to the table every 25 files, and the manifest is written after each batch by writing a temporary file and renaming it over the old one, so a crash mid-run leaves either the old manifest or the new one, never half of one. The first vector’s length is recorded as the index’s dimensions.

Cancel, an error, a save, and a new model

Cancel clears the queue. Files already in flight finish, the pending rows are flushed, the keyword index is built over what was written, and the run returns. The manifest’s updatedAt is not stamped, so the tab does not claim the index is current, but every embedded file is stamped and stays. The notification says how many files were kept, and the next Update carries on from there rather than starting over. A provider error takes the same path with the error shown.

The keyword index is the other thing finishWrites does: a BM25 index over the keywords column, built if missing, with stemming on and stop words removed, followed by a compaction of the table. It is rebuilt only when it does not exist, which is cheap on a table that has not changed.

With twinny.embeddingUpdateOnSave on, which it is by default, a saved file is re-embedded 1.5 seconds after the last save to it, through the same unchanged checks, so saving without changing anything sends nothing. A deleted file is forgotten; a renamed one is forgotten under the old name and scheduled under the new. Saves during a full run are ignored, and only files on disk count, not virtual documents. Nothing happens until the workspace has been indexed once.

Switching the embeddings provider to a different model changes two things. The next Update becomes a rebuild whether or not you asked for one, because vectors from two models do not mix and the manifest says which model made these. And until that run happens, saves stop updating the index: updateFile refuses when the manifest’s model is not the active one. The tab shows the mismatch; the quiet part is that your saves are no longer being indexed in the meantime. Run Rebuild, or switch back.

The settings that matter here are in the tab (chunk sizes of 1,000, 100 and 100 characters by default; changing them needs a rebuild) and in twinny.embeddingIgnoredGlobs. The rest is on the workspace index page at docs.twinny.dev.