Handle an unanswerable RAG question explicitly

An absent handbook rule should produce an explicit unresolved answer. A language model may have enough general knowledge to write something plausible even when the authorized corpus contains no supporting passage. That behavior can make a retrieval system appear more complete than its sources allow.

Build a synthetic handbook containing password-reset instructions but no travel-expense policy. Ask one question about reset steps and another about hotel reimbursement. Label the second question unanswerable from this corpus before running the system.

The original RAG paper combines retrieved information with a generation model. A deployed application's decision to abstain still needs its own policy and evaluation.

Question reaching an empty evidence tray and an explicit unknown response.

Define a useful abstention

A useful response identifies the evidence gap without inventing a policy: "The available handbook passages do not state the hotel reimbursement limit." If an authorized escalation destination exists, provide it. Do not manufacture a contact or assume that another department owns the answer.

Represent the outcome explicitly in the response schema, for example with an answerable flag and an optional explanation. A blank string can be mistaken for a service failure, while a confident paragraph with no source can be mistaken for a supported answer.

A similarity threshold can help reject weak retrieval results, but it is not a universal answerability detector. A highly similar passage may discuss reimbursements without giving a limit. A relevant answer can also score poorly when the question uses unfamiliar wording. Calibrate any threshold using labeled examples from the actual corpus.

Measure both kinds of mistake

Include questions that are fully answerable, partially answerable and absent. Count how often the system invents an answer for absent questions and how often it refuses questions with adequate evidence. Report those outcomes separately, alongside their sample counts.

The scikit-learn metrics guide provides definitions for classification metrics. If you model answerability as a classification task, define which outcome is the positive class before reporting precision or recall.

Partial evidence needs careful wording. A handbook may explain how to request reimbursement while omitting the permitted amount. The answer can state the documented request process and explicitly leave the amount unresolved, with a citation supporting only the process.

This fixture does not establish production reliability. Real questions can contain mistaken assumptions, refer to inaccessible documents or need several sources. Keep a review queue for unresolved cases and distinguish "not found in the retrieved context" from "the organization has no such policy". The search process can miss information that exists elsewhere.