Stopping a call to a function that does not exist
RunEcho checks every edit an agent is about to write against the symbols your code actually declares, and stops the ones that reference something imaginary. It is one cheap layer against one class of mistake — deliberately not an attempt at the whole problem.
Deterministic, so it is a gate and not an opinion
Every check is a parse and a lookup. The same code produces the same answer on every run, every machine and every agent — there is no model to sample from and nothing to re-roll. No API keys, no network, no build, no language server.
A clean check costs zero context tokens, measured rather than asserted: it writes nothing at all unless it actually stops an edit, and then about a hundred tokens. Because it runs as a hook, the agent never spends context deciding whether to call it. The companion server that answers structural questions on demand is a separate surface and is not free — its tool schemas cost around 968 tokens at session start. Both figures, including the unflattering one, are published in the repository.
How often it is wrong, and how that was measured
The obvious measurement is how often the user overrides it. That turned out to have no variance to measure: across thirty days of real traffic, 308 of 308 gated edits were approved and none denied. A rate computed from that would have said the guard was perfect, and it would have been measuring the user’s patience.
So each flagged symbol is judged against dated git history instead — was it defined when the guard objected, and is it defined now:
| Verdict on a flag | Share | Meaning |
|---|---|---|
| Stands | 51.0% | a real reference to something that does not exist |
| Premature | 33.7% | correct, but fired before the symbol existed — an agent writing a caller ahead of its callee |
| False positive | 15.3% | the symbol was there; the guard was wrong |
The 51% is not a claim that half of an agent’s output is hallucinated — it is a share of the edits that were flagged, not of the edits that were written.
The scope, stated before you ask
RunEcho reads unqualified references: bare calls, constant references, and type annotations. Measured against a corpus of real model hallucinations mined from live session transcripts — fifteen hand-verified cases, each backed by a compiler or runtime error as independent ground truth — that catches four of nine.
The other five sit in qualified positions: df.groupby(…),
tree.Root(). Resolving those needs receiver-type inference, which is the
semantic analysis this tool deliberately does not do. Indexing also covers more languages
than the guard checks — shell, Rust and Ruby feed the index but are not validated at edit
time.
That is the honest shape of it: run it the way you run a type checker. One layer that removes one class of mistake completely, alongside the tests and the review that catch the rest.
The design its own dogfood killed
An edit-scope contract declares what a piece of work is allowed to touch, so a guard can object when an agent wanders outside it. The obvious next step was to bind a contract automatically when a session starts. Dogfooding the guard on the day it merged killed that idea: the contract describing the work went stale within the same session, at the moment the work it described was merged.
A session routinely spans several unrelated pieces of work. A contract belongs to a task, not to a session, and between tasks the correct state is no active contract at all — the free, silent, abstaining default. Automatic session binding would have produced a guard that was confidently wrong most of the time, which is worse than no guard.
Choosing a parser by what a shortcut would get wrong
Each language gets a parser engine chosen on one question: are there constructs a cheap length-preserving masker cannot disambiguate without actually parsing? If yes, it earns a real grammar. If no, a grammar is overkill.
| Language | Engine | Why |
|---|---|---|
| Rust | tree-sitter grammar | lifetimes vs char literals; nested block comments |
| Shell | regex + masker | quoting and heredocs resolve in one stateful pass |
The asymmetry is what makes ambiguity the right test. A masker that guesses wrong fails
silently and loses every symbol after the mistake — indistinguishable from a file
with no symbols in it, which is the worst possible failure for a guard. In Rust,
'a is a lifetime in one line and a character literal in the next, and nothing
short of parsing tells them apart.
Visibility follows the same principle rather than a house convention. Go extracts only exported names, because an unexported Go name cannot be referenced from outside its package. Rust extracts everything, because a same-crate reference to a private function is completely ordinary — filtering it out would produce false negatives on the common case.
Guards have to be cheap or they get turned off
Questions about the approach? Email is best — [email protected] · Back to portfolio