Entry 05Developer tooling · Go2026

Stopping a call to a function that does not exist

RunEcho checks every edit an agent is about to write against the symbols your code actually declares, and stops the ones that reference something imaginary. It is one cheap layer against one class of mistake — deliberately not an attempt at the whole problem.

per check~12 ms
context cost, clean check0 tokens
false positive rate15.3%
models, keys, networknone

Deterministic, so it is a gate and not an opinion

Every check is a parse and a lookup. The same code produces the same answer on every run, every machine and every agent — there is no model to sample from and nothing to re-roll. No API keys, no network, no build, no language server.

A clean check costs zero context tokens, measured rather than asserted: it writes nothing at all unless it actually stops an edit, and then about a hundred tokens. Because it runs as a hook, the agent never spends context deciding whether to call it. The companion server that answers structural questions on demand is a separate surface and is not free — its tool schemas cost around 968 tokens at session start. Both figures, including the unflattering one, are published in the repository.

How often it is wrong, and how that was measured

The obvious measurement is how often the user overrides it. That turned out to have no variance to measure: across thirty days of real traffic, 308 of 308 gated edits were approved and none denied. A rate computed from that would have said the guard was perfect, and it would have been measuring the user’s patience.

So each flagged symbol is judged against dated git history instead — was it defined when the guard objected, and is it defined now:

Verdict on a flagShareMeaning
Stands51.0%a real reference to something that does not exist
Premature33.7%correct, but fired before the symbol existed — an agent writing a caller ahead of its callee
False positive15.3%the symbol was there; the guard was wrong

The 51% is not a claim that half of an agent’s output is hallucinated — it is a share of the edits that were flagged, not of the edits that were written.

The scope, stated before you ask

RunEcho reads unqualified references: bare calls, constant references, and type annotations. Measured against a corpus of real model hallucinations mined from live session transcripts — fifteen hand-verified cases, each backed by a compiler or runtime error as independent ground truth — that catches four of nine.

The other five sit in qualified positions: df.groupby(…), tree.Root(). Resolving those needs receiver-type inference, which is the semantic analysis this tool deliberately does not do. Indexing also covers more languages than the guard checks — shell, Rust and Ruby feed the index but are not validated at edit time.

That is the honest shape of it: run it the way you run a type checker. One layer that removes one class of mistake completely, alongside the tests and the review that catch the rest.

The design its own dogfood killed

An edit-scope contract declares what a piece of work is allowed to touch, so a guard can object when an agent wanders outside it. The obvious next step was to bind a contract automatically when a session starts. Dogfooding the guard on the day it merged killed that idea: the contract describing the work went stale within the same session, at the moment the work it described was merged.

A session routinely spans several unrelated pieces of work. A contract belongs to a task, not to a session, and between tasks the correct state is no active contract at all — the free, silent, abstaining default. Automatic session binding would have produced a guard that was confidently wrong most of the time, which is worse than no guard.

Choosing a parser by what a shortcut would get wrong

Each language gets a parser engine chosen on one question: are there constructs a cheap length-preserving masker cannot disambiguate without actually parsing? If yes, it earns a real grammar. If no, a grammar is overkill.

LanguageEngineWhy
Rusttree-sitter grammarlifetimes vs char literals; nested block comments
Shellregex + maskerquoting and heredocs resolve in one stateful pass

The asymmetry is what makes ambiguity the right test. A masker that guesses wrong fails silently and loses every symbol after the mistake — indistinguishable from a file with no symbols in it, which is the worst possible failure for a guard. In Rust, 'a is a lifetime in one line and a character literal in the next, and nothing short of parsing tells them apart.

Visibility follows the same principle rather than a house convention. Go extracts only exported names, because an unexported Go name cannot be referenced from outside its package. Rust extracts everything, because a same-crate reference to a private function is completely ordinary — filtering it out would produce false negatives on the common case.

Guards have to be cheap or they get turned off

Duplicate-symbol warningsBuild-constraint pairs — the same function defined once per platform — produced 91% false positives. Suppressing only when both files are constrained removed all of them, without a toolchain shell-out. A constrained-versus-unconstrained pair still asks.
Store accessChecking the contract meant opening the store a second time, measured at +9.5 ms per edit. Sharing the handle with the symbol lookup already running brought it to +0.36 ms.
Threat modelLocal execution behind a firewall, so no web attack surface — but the binaries are distributed publicly, which puts the release pipeline in scope even though authentication and rate limiting are not.
Developer ToolingGo Tree-sitterAgent Safety

Questions about the approach? Email is best — [email protected] · Back to portfolio