Portfolio
RAG systems get graded on the wrong question.
Most retrieval-augmented generation demos answer one question: does the system respond well when the knowledge base has the answer. That is the easy case. The dangerous failure is what happens when it does not, when someone asks something the documentation never covered and the model has to choose between saying it does not know and inventing something plausible.
This project builds a small RAG system over a fictional SaaS product, then stress-tests exactly that failure with a purpose-built probe set and an independent, cross-family judge, to check whether the system is telling the truth about what it knows.
- Role
- Evaluation design and implementation
- Knowledge base
- Nine markdown documents, about fifteen pages
- Probe set
- Category-split probes, plus a disjoint held-out set
The setup
What is actually being tested.
The knowledge base is nine markdown files covering a fictional SaaS product's pricing, permissions, integrations, API limits, data policy, and billing rules, about fifteen pages in total. Each document is chunked by heading, embedded with Voyage AI, and retrieved by cosine similarity against the query. At this size no vector database is needed. Claude Haiku 4.5 writes the answer from whatever comes back.
The RAG pipeline itself is standard. What this project is about is the probe bank built to fail it on purpose, and the evaluation layer built to catch those failures without trusting itself blindly.
Probe categories
| Probe category | What it tests |
|---|---|
| Answerable | The knowledge base has a direct answer. The baseline case. |
| Soft unanswerable | The knowledge base addresses the question explicitly and the answer is negative, for example that EU hosting is not offered. A grounded answer, even though it is a no. |
| True gap | The knowledge base never addresses the topic at all. Saying so is the only correct answer here. |
| Rephrase pairs | The same question asked two different ways, checking that the system agrees with itself. |
Soft unanswerable and true gap look alike on the surface. Both produce a no or a not-covered response, but they test different things. Conflating them turned out to be the single biggest source of noise in this project's own evaluation design, which is covered further down.
The judge
Finding a hallucination means knowing what the model actually saw.
A judge that grades an answer against general knowledge of the product is really testing whether the answer happens to be true, which is a different question from whether it is grounded. This judge sees only what the generation model saw on that specific call: the question, the retrieved excerpts, and the answer. No gold label, no category, no hint. It picks one of six outcomes, enforced by a JSON schema so that an invalid label is structurally impossible.
Judge labels
| Label | What it means |
|---|---|
| Grounded, correct | Every claim in the answer is backed by the retrieved excerpts and matches them. |
| Grounded, incorrect | The answer cites real content but misapplies it: right document, wrong plan. |
| Fabrication | The answer states something specific that appears nowhere in what was retrieved. |
| Correct refusal | The excerpts genuinely do not cover it, and the model says so cleanly. |
| Incorrect refusal | The model claims not to know, but the answer was right there. |
| Declined with fabrication | A correct refusal at the core, with an unsupported specific claim tacked on anyway. |
The judge runs on a different model family from the generator, so that no model is grading its own homework. It is validated against a hand-labeled gold set using Cohen's kappa rather than raw accuracy, because kappa corrects for how easy the labels were to guess by chance and accuracy does not.
What we found
The interesting part was never the headline number.
Retrieval hits the right document 97.2% of the time within the top four results, across every probe with a known correct source.
Retrieval accuracy
| Metric | Result |
|---|---|
| Hit@1 (correct source ranked first) | 94.4% |
| Hit@4 (correct source in top four) | 97.2% |
That is a clean number, easy to publish and move on from. The more useful finding was the 2.8% that missed entirely. One question, on whether two workspaces can be merged, retrieved four chunks about integrations, roles, and permissions, and none of them was the FAQ entry holding the answer. The response was still accurate and closely matched the real policy, almost certainly because “workspace merging is usually unsupported, contact support” is a common enough pattern that Claude Haiku produced it from general knowledge rather than from anything it had been given. Read the output alone and it looks like a clean, grounded answer. Only cross-referencing the retrieval log against the source of truth exposed that nothing in the response was supported by what the system retrieved on that call. This is the failure that matters most: the right answer for the wrong reason, invisible unless someone goes looking for it.
The judge validation almost lied to us, and the fix is the actual point of this project. The first validation round scored a real disagreement: six probes where a human labeled correct refusal and the judge said grounded, correct instead. Every one of them was a soft-unanswerable probe, and none was a true gap. That split exposed a flaw in the label taxonomy itself. A knowledge base that states a negative fact explicitly should score as a grounded answer, not as a refusal. The definitions were fixed and the affected probes relabeled.
Then came the trap. Re-scoring after that fix produced perfect agreement, a kappa of 1.000 across every category. That number is a symptom of circularity, not of judge accuracy: the answer key had just moved toward the judge's own reading, so agreement was guaranteed by construction rather than earned.
The fix was a second, disjoint probe set: fourteen new questions aimed at the same category boundary, labeled independently and blind to what the judge would later say. That is what restores an actual independent test, and it is where the honest number comes from.
Held-out validation
| Group | n | Agreement | Cohen's kappa |
|---|---|---|---|
| Real generations | 10 | 9/10 | 0.778 |
| Synthetic failure cases | 4 | 3/4 | 0.667 |
| All held-out probes | 14 | 12/14 | 0.794 |
Two disagreements are recorded in the full report, each with its reasoning attached, rather than zero. A perfect score against a well-aligned answer key looks better on a slide. A held-out score with visible disagreement is the one that means something.
Why this matters
This is the differentiator, not the footnote.
Most of what is written above is a story about catching our own mistakes: a labeling taxonomy that conflated two different questions, a circular validation loop, and a case where the right answer turned out to be ungrounded once it was actually checked. None of those surfaced from reading final outputs. They came from deliberately cross-checking retrieval logs against labels, and labels against an untouched holdout set, before trusting any number enough to report it.
That is the standard this kind of evaluation work should be held to on a real production system: would the number survive someone trying to poke a hole in it.
Built with
Python · Claude Haiku 4.5 (generation) · GPT-5.6 Terra (cross-family judge) · Voyage AI embeddings, voyage-3.5-lite · OpenRouter · cosine similarity retrieval, no vector database · Cohen's kappa validation, hand-implemented