Test a RAG system against a fixed question set

A retrieval change is easier to judge when the same questions run against the same source snapshot. A few hand-picked successful answers can hide regressions on names, exceptions or questions the corpus cannot answer.

Create an authored test set before tuning. Each record should contain a question ID, question text, expected source IDs and a short explanation of what those sources support. Keep the corpus revision beside the test-set version so later edits do not silently change the expected result.

For a small synthetic example:

q-reset: How do I reset a password?
expected source IDs: reset-procedure

q-export: When does an export link expire?
expected source IDs: export-retention

q-travel: What hotel rate is reimbursed?
expected source IDs: none; corpus does not answer

The labels describe fixture documents that must exist before the test is meaningful. The final case tests abstention and should not be included as an ordinary positive retrieval target.

Fixed questions and expected evidence feeding a repeatable evaluation.

Define the retrieval metric exactly

A simple hit-at-k measure asks whether at least one expected source appears among the first k results. For answerable questions, count successful hits and divide by the number of answerable questions. State k and the denominator every time.

When a question needs two documents, one hit is too forgiving. Calculate the fraction of required sources retrieved, or label the exact passages needed for complete support. Decide whether overlapping chunks from the same source count once; usually duplicate text should not create multiple successes.

Use the scikit-learn evaluation guide for metric terminology, while documenting application-specific retrieval judgments separately. A retrieval hit does not imply a correct generated answer.

Keep a useful failure report

Save the returned IDs, ranks and source spans for every question. Compare failures by type: a missing source, a poor chunk boundary, an incorrect permission filter or a generator that ignored adequate evidence. Each points to a different next investigation.

Hold back some questions from routine tuning. Repeatedly optimizing the same small fixture can make its score look better without improving unfamiliar questions. Add newly observed failures deliberately, with labels reviewed by someone who understands the corpus.

Changing the embedding model, chunker and prompt at once makes attribution difficult. Run controlled comparisons where possible and record the full configuration for each run.

This is an evaluation design, not a reported benchmark. Synthetic questions make behavior reproducible, but they rarely match the range of real user language. Before publishing quality claims, expand the set with representative authorized questions and report the size and limits of the evaluation.