How to Measure RAG Accuracy Before You Launch

RAG accuracy is not one number, and treating it as one is why so many chatbot projects stall after the demo. A retrieval system can find the right page and still produce a poor answer. It can also write a lovely answer from the wrong page.

So you need two scores, a fixed set of questions, and a gate that blocks a release when either score drops. This guide shows how to build all three without a research team.

Why RAG accuracy is not one number

Retrieval augmented generation has two moving parts. Search finds passages. A model then writes an answer from them. If you have not met the pattern yet, our primer on what RAG means in AI explains the basics.

When an answer goes wrong, you must know which half failed. Otherwise your team tunes prompts for a week while the real problem sits in chunking.

Therefore measure the halves separately. Only then does a single headline number mean anything.

Split RAG accuracy into two scores

First, the retrieval score asks one question: did the right passage reach the model? The answer score asks a different one: given those passages, is the reply true, complete and clear?

Low retrieval and low answer scores mean your index needs work. Good retrieval with a weak answer usually points at the prompt or the model. Good answers with weak retrieval means you got lucky, so do not celebrate.

Build a gold question set first

Of course, everything rests on this set. Without it, you are judging by vibes and by whichever question the CEO tried this morning.

Question type	How many	What it catches
Common questions from real tickets	60	Daily traffic and obvious gaps
Questions with an exception or a condition	15	Passages split at the wrong place
Questions that need two documents	10	Weak reranking and thin context
Questions with no answer in your content	10	Whether the bot refuses politely
Near duplicate questions worded differently	5	Brittle keyword matching

Beside each question, record the ideal answer and the document that holds it. That second column is what makes retrieval measurable at all.

Retrieval metrics that matter

Metric	What it means	Healthy range
Recall at 10	The right passage appears in the top ten	Above 90 percent
Hit rate at 3	It appears in the top three	Above 80 percent
Mean reciprocal rank	How high the right passage sits	Above 0.7
Empty result rate	Searches that return nothing useful	Under 5 percent

Recall at 10 is the number to fix first, because the model cannot use what it never received. Chunk size, hybrid search and reranking all move it. Our note on hybrid search covers the biggest single lever.

Answer metrics that matter

Now judge the reply itself. Four checks cover most of what people care about.

• Grounding. Every claim traces back to a retrieved passage.
• Completeness. The reply includes the exception, not just the rule.
• Correct refusal. When the content has no answer, the bot says so.
• Clarity. A new joiner can act on the reply without asking twice.

Grounding is the one to guard hardest. An ungrounded answer is a confident invention, and one of those costs more trust than ten refusals. We wrote separately about how to stop AI chatbot hallucinations.

Who scores the answers

People score first. Take three reviewers, give them the question, the reply and the ideal answer, and ask for a simple verdict: correct, partly correct, or wrong.

Keep the scale small, since a ten point scale only creates arguments. Also rotate reviewers, because the person who wrote the prompt cannot judge it fairly.

A hundred questions with three reviewers takes about two hours. That is one afternoon a week, and it is the cheapest quality investment in the project.

Using a model as a judge

Manual review does not scale past a few hundred questions. So once your people have scored two rounds, hand the routine work to a model.

Then give the judge model the question, the retrieved passages, the reply and the ideal answer. Ask for a verdict plus a one line reason. The reason matters, since it lets you spot a judge that drifts.

Check the judge against your people every month. If they agree on more than 90 percent of cases, trust it for weekly runs. Otherwise tighten the instructions and try again.

Turn scores into release gates

Gate	Threshold	If it fails
Recall at 10	90 percent	Fix chunking, add hybrid search, rerank
Grounded answers	95 percent	Tighten the prompt, force citations
Correct refusals	90 percent	Add a confidence threshold
Wrong answers	Under 2 percent	Block the release
Change against last run	No drop above 3 points	Investigate before shipping

Write these numbers down before the first test run. Thresholds chosen after seeing the results always drift downward, and everybody knows it.

A weekly loop that protects RAG accuracy

Meanwhile, content changes, models change and prompts get edited on a Friday evening. So run the gold set weekly and compare against the previous score.

Keep every run in a simple table with the date, the change and both scores. Six months later, that table tells you exactly which change hurt, which is impossible to reconstruct from memory.

Our checklist on how to test an AI chatbot before launch turns this loop into a release process.

Watching RAG accuracy in production

Test sets tell you what you thought to ask. Production tells you what people really ask, so log both.

Four signals are enough at the start. Thumbs up and down give a direct read. Repeat questions within one session show weak answers. Hand overs to a person mark the coverage edge. Finally, searches that return nothing name the missing content.

Also review the worst twenty conversations every week. Add the interesting ones to the gold set, and your test set slowly becomes a mirror of real traffic.

Mistakes that inflate your score

Some habits make the numbers look great while users stay unhappy.

• Writing questions from the documents. They match too well and hide real gaps.
• Dropping hard questions. The ones that fail are the ones worth keeping.
• Counting a partly correct answer as correct. Half a policy is a wrong policy.
• Testing on a frozen index. Real content changes every week.
• Never testing refusals. Then the bot answers everything, including what it should not.

What good numbers look like

For an internal knowledge assistant on reasonably clean content, expect recall at 10 above 90 percent, grounding above 95 percent, and wrong answers under 2 percent after a few rounds of tuning.

First runs look worse. A recall score near 60 percent on day one is normal, and it usually climbs fast once chunking and reranking get attention. Still, judge progress against your own baseline rather than against a vendor slide.

Chunking is the biggest lever on RAG accuracy

Most teams reach for a better model when the score disappoints. Chunking usually pays more, and it costs nothing but an afternoon.

Split documents at their headings rather than at a fixed character count. A rule and its exception then stay in the same passage, so the model sees the full picture.

Keep passages between 300 and 800 words for policy content. Shorter pieces lose context, while longer ones bury the answer in noise. Also carry the document title and section name into every passage, because those few words lift retrieval more than any prompt tweak.

Re-run the gold set after each chunking change. You will often see recall jump ten points in one step, which no prompt edit will ever match.

Who owns RAG accuracy in your team

Projects drift when nobody owns the number. So name one person, usually the product owner rather than an engineer.

That person runs the weekly evaluation, reviews the worst twenty conversations, and decides whether a release ships. Engineers fix what the report shows, while content owners fix the documents behind the failures.

Finally, share the score openly with the wider team each week. Once people see RAG accuracy move with their own content edits, they start writing better documents without being asked.

What an evaluation costs

The bill is small, which surprises most finance teams.

A weekly run over a hundred questions costs a few hundred rupees in model calls. Human review takes two hours, and the setup takes a day or two once the gold set exists.

Still, compare that with the cost of a wrong refund rule reaching five thousand people. Evaluation is the cheapest insurance in the whole project.

How Intellowork measures RAG accuracy

Intellowork ships with an evaluation workspace, so the gold set, the weekly runs and the score history live beside the index itself.

Every answer stores its sources, which makes grounding checks automatic rather than manual. Teams see the score per department too, since HR content is often clean while field SOPs are not.

If you would like to see your own RAG accuracy measured on your own documents, talk to our team. We usually run a first evaluation within a week.

Frequently asked questions

How many questions does a gold set need?

A hundred well chosen questions beat a thousand random ones. Add ten each week from real traffic, and the set stays useful for years.

Can we measure RAG accuracy without ideal answers?

Partly. You can score grounding and refusals without them, yet completeness needs a reference. So write the ideal answers for at least the top fifty questions.

How often should we run the evaluation?

Weekly, plus before every release. Anything less and a quiet content change will surprise you in front of users.

Is a model judge reliable enough?

Yes for routine runs, once it agrees with your reviewers on more than 90 percent of cases. Keep a monthly manual check as a control.

What if RAG accuracy drops after launch?

Look at content first, since most drops follow a document change or a new folder. Then check the index, and only after that the prompt.

Does a higher model always improve the score?

No. When retrieval is weak, a bigger model writes a more fluent wrong answer. Fix search first, because that is where the cheap gains are.

The short version: measure retrieval and answers separately, write your gates before you see the numbers, and run the same questions every week. RAG accuracy then stops being a debate and becomes a dial your team can turn.