Building Retrieval Systems That Refuse to Guess

Demos reward confident answers. Production punishes them. The first time a support assistant invents a refund policy, you lose more trust than the previous thousand correct answers earned. So we design backwards from the failure: what does the system do when it does not have the answer?

Content quality beats model choice

Before touching embeddings, consolidate the source material. Duplicate policies with different dates, a help centre contradicting a PDF, three years of decisions living only in chat - no retrieval strategy survives that. Retrieval quality is a content problem first and a vector problem second.

Citations are not decoration

Every claim in a generated answer should point at a passage a human can open. This does two things: it lets a reviewer verify the answer in seconds, and it makes hallucination structurally awkward - the model has nothing to cite when it invents.

• Hybrid retrieval: keyword plus vector, then rerank. Pure semantic search misses exact identifiers.
• Metadata filters: region, product line, effective date. Most wrong answers are right answers from the wrong context.
• Confidence gate: below a retrieval score threshold, escalate to a human instead of generating.
• Golden set: real historical tickets, scored on every deploy. Prompt changes are code changes.

Escalation is a feature

The systems that get trusted are the ones that hand over cleanly - with the conversation, the retrieved context, and the reason for the handoff attached. Agents stop resenting the bot when the bot makes their job shorter instead of messier.

“An assistant that says 'I don't know' 15% of the time is worth more than one that is confidently wrong 5% of the time.”