Reranking Retrieved Evidence Before Asking the LLM

Give the second stage a bounded job

The initial retriever should find a reasonably broad set of eligible candidates. The reranker then focuses on relevance to the question. It cannot recover a missing document, repair failed extraction or authorise a source that the user is not allowed to see.

A cross-encoder processes a query and passage together, unlike independently generated embeddings used for an initial vector search. This can justify a more expensive comparison on a small shortlist. The number of candidates and passage length therefore belong in the latency and cost budget.

Keep source identity attached to every score

The example illustrates deterministic ordering after a hypothetical reranker has returned scores. The scores are invented. A real integration needs model input limits, provider failure handling and validation that every returned score corresponds to exactly one authorised candidate.

candidates = [
    {"source_id": "email-a", "score": 0.42},
    {"source_id": "email-b", "score": 0.87},
]
ranked = sorted(candidates, key=lambda row: (-row["score"], row["source_id"]))
assert ranked[0]["source_id"] == "email-b"
assert {r["source_id"] for r in ranked} == {"email-a", "email-b"}

Run an ablation before adding a dependency

Compare the existing retriever with the same retriever plus reranking on fixed, labelled questions. Measure whether supporting passages move into the answer context, how many relevant passages are lost and how much latency is added. Evaluate citation support downstream, since better ranking is only useful if the answer benefits.

Define the fallback explicitly. If reranking fails, the product may use the original eligible shortlist with an appropriate status rather than fail the entire search. The decision depends on measured quality. A reranker should earn its place through evidence, not because a more elaborate retrieval diagram looks more advanced.