Experiment 08
The alarm that cried wolf every day
My knowledge base has an automated quality check: 156 search queries with known right answers, run before every code change, so I cannot accidentally make search worse. One problem. The collection grows every day, and growth alone made the scores drop. The alarm went red when I broke something and red when I simply added notes, which teaches a person to ignore alarms.
The fix was to freeze the exam. The check now scores the same fixed set of questions against the same fixed pool of candidate answers, captured on the day the baseline was stamped. New notes are still searchable; they just do not get graded. Growth stopped moving the score at all.
Then I proved the alarm still works: I deliberately sabotaged the ranking and ran the check. The score collapsed and the gate failed the change, exactly as it should. Passing growth is only half the job; an alarm that passes everything is as useless as one that fails everything.
The check has run before every commit since. Red now means broken, not bigger.
The instrument, verified
The diagnosis was retrospective and clean: historical corpus states rebuilt from ingestion timestamps, same embeddings, same code, only the distractor pool varying. The score drop tracks pool size alone. The alarm was measuring growth, not damage.
The freeze pins two things at the moment a baseline is stamped: the query set and the pool of candidate documents that gets scored. New notes remain fully searchable in production; they simply do not join the graded pool until a new baseline is deliberately stamped. Scores become comparable across months.
The acceptance run is the proof the alarm still works: frozen scoring on the grown corpus comes back green, and the same run with a regression deliberately injected into the searchers comes back red, against a tolerance of 0.02 on hit@51.
The gate has since paid rent as an instrument, not just a guard: the native-SAE experiment rebuilt the ranking in a mirror that reproduced this baseline exactly, then used the gate to reject a reconstructed index at every configuration tried. An eval you can hand to another experiment as a trusted ruler is the difference between a test suite and an instrument.
- hit@k: the fraction of eval queries whose one correct document appears in the top k results. ↩