We Built a Neural Reranker and Then Threw It Away
A cross-encoder reranker is the standard next step for a RAG pipeline. We evaluated one on a real retrieval product, it confidently ranked the wrong documents first, and the replacement was a scoring function you can read in one screen.
The standard advice for improving a retrieval-augmented generation pipeline goes like this: once basic vector search works, add a cross-encoder reranker. Let a neural model re-score your top candidates against the query, and relevance improves. It is good advice often enough that it has become a default.
We followed it on a retrieval product we’re building, a legal-procedure knowledge base where the retrieved text has to be the right rule, for the right state, ranked for the person actually asking. We had the reranker model already sitting on the box. We wired up the evaluation. And then we rejected it, because on the discriminations that actually decide whether our product works, it was confidently wrong.
The problem
Our pipeline had two ranking problems that plain similarity search could not solve.
The first was scope. A user asking about a Texas deadline should get a refusal, not a Florida answer dressed up as one. But embedding similarity cannot see that. In our tests, an out-of-scope query about another state’s deadline scored higher cosine similarity than a legitimate in-scope question, because everything about the query matched except the one word that mattered.
The second was audience. Our corpus contains both plaintiff-side form templates and defense-side guidance. Court forms carry high formal authority, so a naive authority-weighted sort kept floating the plaintiff’s complaint template above the defense guidance a defendant actually needed. The top result was authoritative, on-topic, and useless to the person asking.
A reranker is the textbook fix for exactly this kind of nuance. So we tested it.
The analysis
The model was a 4-billion-parameter cross-encoder reranker, run locally, returning usable yes/no relevance judgments with log probabilities. Mechanically, it worked fine. The failures were in the judgments.
With a jurisdiction-aware relevance prompt, it scored the out-of-scope other-state query at 0.90. It scored a plaintiff form template at 0.92 against a defense-side query. Those are not marginal misses. Those are high-confidence rankings of precisely the documents our two known bugs surface, which means deploying it would have made both problems worse while adding a model call to every query.
There was a second cost that took longer to articulate. Our product’s whole thesis is that every answer traces to text you can point at. A neural reranker inserts an unexplainable judgment into the middle of that chain. When a user asks why result A outranked result B, “the 4B model preferred it” is not an answer we can stand behind, and in this domain, being able to stand behind the ranking is the product.
The solution
What replaced it is almost embarrassingly simple: a transparent composite score, a few lines of arithmetic over properties we already store on every chunk.
Each result’s final score is the fused retrieval score, plus a bounded adjustment for audience weight (defense-side guidance weighted up for this product’s users, form templates weighted far down unless the query explicitly asks for a form), plus a small bonus for primary authority, minus a penalty for jurisdiction mismatch. Scope enforcement stayed where it already worked, in a metadata filter plus a small lexical guard for explicitly out-of-scope requests. On top of that sits a bounded coverage guarantee: a question-shaped query is guaranteed its best direct-answer chunks in the final set, capped at a few swaps, so a weakly-similar but on-point answer is never buried.
Every term in that score is inspectable. We expose the per-term breakdown, so “why did this rank here” has an exact answer.
The evaluation harness that killed the reranker also validated the replacement: after the rebuild, the full suite passed, including the retrieval, jurisdiction-confusion, and refusal dimensions the reranker would have degraded, with no added latency, because the ranker is pure arithmetic over a candidate pool of forty instead of a second model call.
Lessons
Evaluate the component on your failure cases, not on the benchmark’s. The reranker is presumably fine at what it was trained for. Our discriminations, one state versus another, one audience versus another, are not general relevance, and the only way to learn that was a harness built from our own known failures. The evaluation cost an afternoon. Shipping the reranker and discovering this in production would have cost a lot more.
“Add a reranker” is a hypothesis, not a step. Earlier in this build we had written down that a reranker was empirically necessary, based on a real observation that similarity scores could not separate out-of-scope queries. The observation was right; the conclusion was wrong. The fix for that specific failure turned out to be a filter and a guard, not a smarter black box. It is worth noticing when a planned component is solving a problem you have since solved another way.
Structure you already have beats intelligence you have to rent. The composite ranker works because the corpus is annotated: every chunk knows its jurisdiction, its document type, its audience. That metadata was expensive to curate, and the payoff is that ranking becomes arithmetic instead of inference. This is the same lesson as our earlier pipeline posts from another angle: the data layer is where the work is, and when the data layer is good, the clever model on top often turns out to be optional.