Hybrid Search and Reranking

Your system answers natural-language questions correctly, but consistently misses three categories: exact references like MNT-2019-07, internal acronyms, and rare proper nouns. You've tried a larger embedding model, a higher dimension, a finer chunking strategy. Recall moves by two points. These questions account for 30% of real traffic, because users who know a document's reference type it in instead of describing what they're looking for.

No embedding model will fix this, for the reason covered in lesson 3: an alphanumeric reference has no learned semantics, so its vector is noise. A lexical index, on the other hand, finds it immediately, because that is exactly its job: matching rare strings. Conversely, the lexical index fails where vector search excels: the question that shares no words with the answer. The two methods don't compete, they complement each other, and the interesting question is how to fuse them.

BM25 in Thirty Lines

BM25 ranks documents by the frequency of the query terms, weighting them by their rarity in the corpus and dampening the effect of document length. You can get it from PostgreSQL with ts_rank, or implement it in memory.