Experiment 24
Teaching my search index to explain itself
My search engine stores every note as a long list of numbers that no human can read. It works, but it cannot tell me what it thinks a note is about. I wanted a translation layer: the same space, but with names.
I trained a small dictionary model1 directly on those production numbers, 16,483 of them covering about 5,000 notes. The entries it learned are strikingly concrete: "icon library migration", "multi-agent systems for parallel tasks", "system design interview preparation". These are real threads in what I save and build, recovered from raw numbers with no labels given. Half of the strongest entries showed up again when I retrained from scratch, so they are not one run’s accident.
Then the honest part. I rebuilt the search index through this readable version and ran my own quality gate on it, the same automated check every search change must pass. It failed, clearly, in every configuration I tried: about one in five of the top results goes missing. So the verdict shipped exactly as measured: a readable side channel beside the search engine, not a replacement for it.
I would rather have a translation layer that admits its cost than a pretty demo that hides one.
Pricing the readable index
The discipline first. 16,483 vectors, split by note key so a video's segments never straddle train and test. The sweep says small: a 2048-latent dictionary at k=32 reaches 0.825 held-out reconstruction cosine, and doubling the dictionary makes things worse, overfitting 15k training vectors with a larger train-validation gap and an earlier peak. Dead latents are a non-issue; the auxiliary loss keeps over 99.9% alive.
Before measuring the SAE's cost, the ruler itself was verified: a numpy mirror of the production ranking reproduced the frozen eval baseline exactly, hit@1 .7115, hit@5 .9038, hit@10 .9423, and every per-bucket cell. Only then was the index swapped for reconstructions. Every configuration fails the gate: hit@5 drops by 0.026 to 0.081 against a tolerance of 0.02, and one in five top-10 results changes. The subtler finding: reconstruction cosine and retrieval faithfulness rank the configurations differently, so a cosine-only sweep would have picked the wrong dictionary.
Are the features real? The dictionary at large is not reproducible: across independent runs only about 9% of latents find a counterpart above 0.8 cosine, and the number is identical whether the data split moves or not, which pins the instability on the optimizer, not the data. The frequently-firing head is different: restricting to the top-100 latents by firing rate and demanding a match against both independent runs gives a mean best-match of 0.762, with half above 0.8. The named features are largely real directions; the long tail is fit noise.
The taste experiment ended in an honest null with a lesson attached. Predicting deliberate saves worked at AUC2 0.771, but the raw embedding solved the same task at 0.978, and the reason is a confound: every deliberate-save in this corpus is Instagram, TikTok or web, and every unrated note is YouTube. The label is the medium. Holding the medium fixed leaves 27 and 19 positives, too few for any feature to keep a stable sign. The output of the experiment is that this corpus cannot currently support the question.