Eric Minish Personal site
Entry 03Retrieval · evaluation2026

My semantic search worked at 1,000 documents and failed at 57,000

I built a hybrid retrieval engine over years of my own working notes, measured it honestly, and the measurement said half of it does not work. This is that result.

corpus57,500
useful answers5 of 10
dev corpus, semantic0.731
dev corpus, keyword0.436

The result that argued against my own design

On a 1,271-chunk development corpus the semantic arm clearly beat keyword search on paraphrased questions — the ones where you describe a memory in different words than you originally used. That is what justified building a hybrid at all.

Measured at full scale for the first time, against ten real questions judged by hand, it returned something useful five times. Every success was a keyword-shaped question. Four of the five paraphrased questions — the exact case the semantic arm existed to solve — returned nothing useful.

ArmRank of the correct answer · 5 paraphrase questions
Keyword (BM25)3 · 1 · 1 · 5 · 1
Semantic>200 · 7 · 2 · 107 · 1

The answers were being retrieved. The semantic arm was burying them. Keyword search alone beat every hybrid configuration — the exact inverse of the small-corpus result. Similarity stops discriminating as the candidate pool grows; rare words stay rare.

How it got there

  1. Small corpus, clear win. 1,271 chunks. Semantic beat keyword on paraphrase, decisively.
  2. Real scale, five of ten. Judged by hand, no partial credit.
  3. The obvious explanation was wrong. "The bigger corpus is diluted" — tested by restricting the search to the single highest-quality source. No improvement. Rejected in one run rather than argued about.
  4. Recall or rank? A diagnostic separated answers that were never retrieved from answers that were retrieved and then buried. It was rank.
  5. The tuning already shipped was a regression. A query-adaptive router — use the expensive re-ranker only when the query looks keyword-shaped — had been tuned on the small corpus and shipped as the default. At full scale it measured worst of six configurations, scoring 0.000 on paraphrased questions where always re-ranking scored 0.200. Removed as default the same day.
  6. Then the evaluation itself had a bug. A code review found the scorer retrieving deeper than the judging pool went, so the re-ranked configuration was returning documents nobody had graded and being marked wrong for it. Fixing it improved that configuration, 0.352 to 0.447 — it did not rescue the conclusion.

The parts that are actually rare

Pre-registered kill gateWritten before anything was built: if a local re-ranker does not beat the existing baseline, the project stops at one session.
Refused to fit anecdotesA visibly underperforming threshold was left alone rather than re-tuned against eight unlabelled examples. Twice — and the refusal is recorded in the plan file both times.
Negative controlsTwo questions with no answer anywhere in the corpus. The semantic arm scored a nonsense query about sourdough hydration 95% as highly as a real question. Its confidence carries almost no information.
Pooled, blind judgingCandidates unioned across five retrieval configurations and shuffled, with no indication of which system found them — so the incumbent could not win by being graded only on its own results.
The evaluator as a defectLabels are stamped with who produced them, and a separate tool samples them to compute inter-rater agreement before they are trusted.
The most damning findingFor several failed questions the answer sat in a file the index demonstrably returns for other questions. Not a coverage gap — a failure to connect a paraphrased need to a document it already holds.
RetrievalEvaluation PythonOffline / CPU

Questions about the method? Email is best — [email protected] · Back to portfolio