Add a reranking stage to a small retrieval pipeline

A retriever can cheaply find plausible passages while leaving several weak candidates near the top. A reranker spends more work comparing the question with each candidate and produces a new order. It can only evaluate passages that the first stage retrieved.

For a synthetic handbook query about cancelling an export, create candidates for starting an export, cancelling one and deleting an account. Save the original order and exact passages. The cancellation passage should be the relevance target if it contains the required procedure.

The Sentence Transformers retrieve-and-rerank guide describes a bi-encoder retrieval stage followed by a cross-encoder that scores query-passage pairs.

Retriever candidate set narrowing through a reranker.

Preserve both candidate orders

Keep a result record containing candidate ID, initial retrieval rank, reranker score and final rank. Without the initial order, a changed answer can be difficult to attribute to the new stage. Also retain the model configuration and source revision used for the experiment.

Choose the first-stage candidate count separately from the number of passages shown to the answer generator. A wider candidate pool gives the reranker more material to inspect, but it costs more processing. The right size depends on observed recall and the application's response-time budget.

Use a test question whose correct passage is absent from the first stage. The reranker must not receive credit for fixing that case unless retrieval actually changes. Reordering the wrong candidates still leaves the answer unsupported.

Check truncation and duplicate passages

A query and a long passage may exceed the selected model's supported input length. Check the model's actual tokenizer behavior and limits before using it. If the cancellation instruction appears only near the end of a truncated passage, its score can be misleading.

Overlapping chunks can also occupy several final positions with nearly the same text. Group or deduplicate candidates using source spans when appropriate, then inspect whether useful supporting passages have been pushed out.

Reranker scores require interpretation specific to the chosen model and configuration. Do not treat every score as a calibrated probability that the answer is correct. A threshold copied from another model can behave very differently on your corpus.

This example describes a reviewable experiment; no cross-encoder was run and no improvement is claimed. Start with a small fixed question set and compare the final evidence selected for each question. Measure both useful rank changes and added latency before deciding whether the stage belongs in the request path.