How to Test a RAG Application Before Production

Your RAG application answers five questions correctly. The demo looks good. Somebody says it is ready for production.

Then a real user asks about a policy that changed last Tuesday, another asks a vague follow-up, and a third gets an answer from a document they should never have been allowed to see.

Lovely. The demo did not include any of that.

RAG testing checks whether retrieval augmented generation (RAG) finds the right evidence, produces an answer supported by that evidence, and behaves sensibly when something goes wrong. Before launch, test retrieval and generation separately, then test the complete application under realistic scenarios. Include permissions, missing answers, latency and cost from the start.

This guide covers the test cases, retrieval evaluation, generation checks, automated metrics and release process I would put in place before letting a RAG application loose on real users.

Why RAG Testing Needs More Than a Good Demo

RAG systems can fail at several different points and still produce a beautifully written final response. That is part of the problem. Fluency makes a bad answer look finished.

The retriever might miss the correct document. It might find the right document but select a chunk that lost the important exception. Or it might retrieve perfectly good context and watch the model ignore it.

Those failures need different fixes. Swapping the LLM every time an answer is wrong is an expensive way to avoid looking at the retrieval layer.

Traditional QA still matters. Authentication, API responses, output schemas and access controls need explicit pass/fail checks. But generation can produce several valid answers to the same question, so an exact string comparison is often the wrong test.

Quality engineering for retrieval augmented generation systems needs both approaches: deterministic checks where the behaviour must be exact, and scored evaluation where wording can vary. A refund amount should be correct. The sentence explaining it can survive a paraphrase.

If you want to diagnose existing failures first, my guide on how to improve RAG performance covers the usual suspects and what to inspect before changing models.

Map the RAG Pipeline Before You Test It

Write down the actual process your application uses. RAG pipelines usually include document ingestion, cleaning, chunking, embedding, indexing, retrieval, optional reranking, prompt construction, generation and citation formatting.

You need to know what each stage receives and what it returns. Otherwise, root cause analysis becomes a meeting where everybody has a theory and nobody has the retrieved chunks.

I would test in three layers:

1. Retrieval only. Given a user query, does the search return the expected evidence in a useful order?

2. Generation with fixed context. Given known good passages, does the model produce a correct, grounded answer?

3. End to end. Does the complete system work for the actual user, with their permissions, conversation history and operational constraints?

The fixed-context test is particularly useful. If the answer is still wrong when you hand the model the right evidence, you have narrowed the problem considerably.

My RAG pipeline guide explains the full flow from ingestion to generation. The RAG architecture guide covers how the components fit together in production.

Step 1 Define What Passing Actually Means

Before running a RAG evaluation, decide which failures should block a release. Do this while everyone is calm, rather than five minutes before launch when the score looks inconvenient.

Define expectations for retrieval accuracy, answer correctness, grounding, safety, latency and cost. Tie each one to the job the application performs.

An internal documentation assistant and a system supporting decisions in highly regulated industries should not share a quality gate because somebody copied the same checklist. The consequences are different.

A useful gate might say that the new configuration must not lose the expected evidence on critical policy questions, must return the required structured fields, and must stay within your agreed p95 latency and request budget. Permission leaks should block release even if the average answer score improves.

Compare against a baseline, inspect results by query category, and repeat borderline generation tests when sampling could explain the difference. Document whether a change is measured in percentage points or relative percent. They are not interchangeable.

Avoid pretending that a particular recall score or a two-second response time is universally production-ready. Choose thresholds from user expectations, observed performance and the risk of getting the answer wrong.

Also, passing every safety probe in your suite means those probes passed. It does not mean you have mathematically defeated every possible attack. Sadly, the internet keeps inventing new ones.

Step 2 Build a Golden Test Set from Real Questions

A golden test set is a collection of queries with known relevant documents and expected answer behaviour. Without one, every experiment becomes a fresh opportunity to convince yourself things improved.

Start with real questions from support tickets, FAQs, internal searches or a small pilot. Protect personal and confidential data when preparing the datasets. Then add the awkward cases your demo politely avoided.

For each test case, store:

• A stable ID, the user query and any necessary conversation history.

• The user or role being simulated and the documents they may access.

• Expected source IDs or passages, with relevance labels where useful.

• Required facts, acceptable answers and expected clarification or refusal behaviour.

• The query category, risk level and corpus version.

A reference answer helps, but it should not force the model to repeat one particular sentence. Describe the facts that must be present and the claims it must not make.

Fifty to two hundred well-chosen queries can be a practical starting point. That is a planning range, not a magic statistical guarantee. Coverage matters, rare failures need targeted tests, and bigger releases may justify much larger suites.

Keep a development set for prompt tuning and a separate held-out set for release evaluation. If you repeatedly tune against every question in the golden set, you may just get very good at answering your own exam.

Synthetic queries can help fill gaps. Review them against the source material and actual user language. A test set written entirely by a model often contains unusually cooperative users. Real users did not receive that memo.

Step 3 Evaluate Retrieval Quality Before Generation

Run retrieval evaluation without asking the LLM to generate anything. Inspect the returned chunk IDs, text, metadata, scores and rank positions.

The first question is whether the relevant documents appeared at all. The next is whether the passages contain enough information to answer correctly. Finding the right PDF does not help if your chunk stops immediately before the exception.

Use a few metrics that explain different failures:

• Recall@k measures the fraction of labeled relevant items found in the top k results.

• Hit rate@k measures the fraction of queries with at least one relevant result in the top k.

• Precision@k measures how many of the top k results are relevant, helping expose poor retrieval precision.

• Mean reciprocal rank, or MRR, rewards placing the first relevant result near the top.

• Normalized discounted cumulative gain, or nDCG, evaluates ranking with graded relevance, rewarding more useful results at higher positions.

Recall and hit rate are easy to confuse. If a question needs four relevant chunks and retrieval finds only one, recall is 25%. Hit rate for that query is still a success. They coincide when each query has exactly one relevant item, which is why this mistake survives so many blog posts.

For context precision and context recall, record how your chosen evaluation framework calculates them. Some implementations use passage labels; others judge whether retrieved context supports claims in a reference answer. Scores with the same name are not automatically equivalent. The Ragas context recall documentation explains this distinction, and Stanford's information retrieval guide covers ranked retrieval metrics.

Measure the initial candidate pool and the context that actually reaches the prompt. A search returning the right chunk at rank 17 can look healthy until your prompt builder keeps only the top five.

Compare chunking strategies, filters, search settings and embedding models while holding the other variables steady. Test exact product codes and error IDs as well as natural-language questions. My RAG embeddings guide explains how to compare models on your own retrieval task.

Better retrieval quality gives generation better evidence. It does not guarantee a better answer. That is why the next test exists.

Step 4 Test Generation with Known Good Context

Give the generation stage fixed, verified passages and assess the LLM response against a clear rubric.

Check factual correctness, faithfulness, answer relevance and completeness. Correctness asks whether the answer is true for the task. Faithfulness asks whether its claims are supported by the supplied context. A model can faithfully quote a stale policy and still give the wrong current answer.

Answer relevance checks whether the response addresses the question. Completeness checks whether it includes the conditions, exceptions or steps the user needs. Then check citations, required formatting and tone.

For example, suppose the context says enterprise customers may cancel within 30 days, except where a signed contract specifies a different term. An answer saying “Everyone can cancel within 30 days” has dropped the bit most likely to matter. It can sound excellent while failing the test.

Use exact assertions for dates, amounts, identifiers and structured fields. For open-ended explanations, use reference facts and an explicit scoring rubric rather than insisting on an exact paragraph match.

Also test missing and conflicting context. Does the model ask for clarification, explain the conflict, or say it cannot determine the answer? A safe no-answer response can be the correct output. For answerable queries, unnecessary refusal is a failure too.

Prompt engineering can help specify these behaviours. It cannot repair evidence that never reached the model. Test prompt changes against fixed context first, then rerun the complete pipeline to see whether generation quality actually improved.

Step 5 Cover the Test Cases Users Will Actually Create

Your end-to-end suite should include realistic scenarios, not just questions copied from document headings.

I would cover these categories:

• Simple lookups with one clear source, checking the answer and citation.

• Multi-document questions that require all the relevant evidence, not just one plausible passage.

• Ambiguous queries and vague follow-ups that need conversation context or clarification.

• Questions with no answer in the corpus, where guessing must not pass.

• Exact terms, prices, dates, versions and numerical calculations.

• Stale, deleted, duplicated and conflicting documents.

• Different languages, typos and domain-specific wording where users need them.

• Permission boundaries between tenants, workspaces, users and roles.

• Prompt injection in both user input and retrieved documents.

• Timeouts, empty search results and failed dependencies.

Make the expected behaviour specific. “Handle ambiguity well” is difficult to test. “Ask which product the user means before quoting a cancellation policy” is much more useful.

For permissions, inspect the retrieval result and assembled prompt as well as the visible answer. Private data should not reach the model just because a later instruction tells it to keep quiet.

If your application includes agents that search repeatedly or call tools, test those actions too. Check source selection, loop limits, tool inputs and downstream effects. My agentic RAG guide explains when those extra retrieval steps are useful and why they need visibility.

Evaluate observable reasoning through evidence use, calculations, citations and tool behaviour. You do not need access to a model's hidden chain of thought to notice that it used the wrong policy.

Automate Scoring Without Letting the Judge Mark Its Own Homework

Manual testing is valuable, but reading every answer after every change becomes a full-time job surprisingly quickly.

Automate retrieval metrics, schema checks, required facts, latency and cost. Use an LLM judge for checks that need interpretation, such as whether an explanation is grounded and addresses the query.

Give the judge a specific rubric, the user question, the answer and the evidence it needs. Reference-free scoring means there is no gold answer; it does not mean the judge can assess grounding without the retrieved context.

Ragas is an open source framework with a faithfulness metric that checks whether response claims are supported by context. Whatever tools you choose, inspect metric definitions and test them against examples your reviewers already understand.

Keep the judge configuration stable: model identifier or version where available, prompt, rubric, sampling settings and evaluation code. Record provider changes when exact model snapshots are unavailable.

Human calibration matters. Have reviewers independently score a varied sample, compare their decisions with the automated scores, and investigate disagreement. Include subtle errors, missing exceptions, unsupported citations and answers that are fluent but wrong.

A judge can reward verbosity, miss factual mistakes or change its scoring behaviour. Human review should stay in the loop for critical cases, borderline releases and unfamiliar query categories.

Automated metrics make evaluation repeatable. Human feedback helps establish whether you are repeatedly measuring something useful.

Test Latency Cost and Failure Handling

An accurate answer that takes 25 seconds can still be a terrible support experience. An accurate answer that triggers six unnecessary model calls can also become a budget problem at scale.

Measure time to first token separately from total response time. Track p50 and p95 latency under realistic concurrency, not just one request on an otherwise idle staging system.

Break duration down across query rewriting, embedding, retrieval, reranking, generation and external tools. My vector databases for RAG guide covers retrieval infrastructure choices and the trade-offs worth testing.

For cost, include model calls, retrieval services, retries and any judge calls used in ongoing evaluation. Report cost per request and per 1,000 queries at an expected workload. Keep indexing and re-embedding costs visible separately.

Force failures deliberately. What happens when search times out, the reranker is unavailable or the model returns invalid output? Check bounded retries, useful error handling and safe fallbacks. “Retry forever” is not a recovery strategy. It is a subscription to the same failure.

Test caches for tenant isolation and freshness. A fast answer from another customer's cached context is not an optimization you want to celebrate.

Put Regression Testing into the Release Process

Run the same evaluation when you change the corpus, chunking, embedding model, retriever, reranker, prompt or generation model. Changing documents is a system change even when nobody edits application code.

Use a small smoke suite for quick feedback on critical paths. Run a broader regression testing suite regularly, then a release evaluation covering important query categories, security checks, latency and selected human review.

Keep a baseline and compare the same test cases under equivalent conditions. Report aggregate results alongside failures by category. An improvement on easy FAQs should not hide a regression on permission checks or current policy questions.

Use version control or a versioned data store for test cases, corpus snapshots, index configuration, prompts, model settings and judge rubrics. Log the configuration with each run so you can reproduce the important parts of a failure.

A CSV file can be enough to start reporting results. Include the case ID, expected sources, retrieved IDs, answer, scores, timings, cost, configuration version and failure reason. Protect sensitive content in reports and traces.

You do not need a massive evaluation dashboard before you have useful tests. You need a report that tells somebody what broke and enough evidence to fix it.

Keep a Feedback Loop Running After Launch

Pre-production testing covers the scenarios you know to test. Live traffic introduces the rest.

Start with a limited rollout, monitor failures, and define when you will pause or roll back. Add production monitoring for retrieval quality, generation quality, latency, cost and access-control failures.

Collect user feedback through corrections, answer reports, support escalations and sampled reviews. A thumbs-down is a useful signal, but it does not explain the failure on its own. Look at the trace.

When a meaningful failure occurs, reproduce it, identify the cause and add a regression case. Refresh relevance labels when policies or documents change. Keep fresh held-out queries so the suite does not become a museum of bugs you already fixed.

Continuous monitoring should lead to action. If a dashboard turns red and nobody owns the response, you have built a decorative alarm.

Where Fetch Hive Fits into RAG Testing

This is also why I care about tracing in Fetch Hive. When a workflow gives a bad answer, I want to inspect model calls, tool calls, duration, usage, cost and failures. Staring at the final response gives you very little to work with.

Fetch Hive helps you build, test, deploy and trace agents and AI workflows. Use that visibility alongside an evaluation framework suited to your application. You still need labeled test cases, sensible metrics and a human definition of what good looks like.

For a concrete example of evaluation uncovering failures that a smaller suite missed, I wrote about what happened when we doubled the workflow compiler eval set. The same lesson applies to RAG: passing the questions you already know is only part of the job.

A Pre Production RAG Testing Checklist

Before launch, I would want clear answers to these questions:

• Have we mapped the pipeline and tested retrieval, fixed-context generation and end-to-end behaviour?

• Does the test set include real questions, edge cases, ambiguous queries and missing answers?

• Do we know which relevant documents each query should retrieve?

• Have we checked grounding, correctness, answer relevance and citations separately?

• Are permissions enforced before restricted context reaches the model?

• Have we tested stale data, prompt injection, dependency failures and safe fallbacks?

• Are latency and cost acceptable under realistic load?

• Is the LLM judge calibrated against human review?

• Can we reproduce runs and detect regressions before release?

• Do we have a rollout plan, production monitoring and a feedback loop that creates new tests?

If several answers are no, that gives you the next work to do. Buying a larger model will not quietly complete the checklist for you.

The Takeaway

Good RAG testing tells you whether the application found the right evidence, used it correctly and behaved safely when the evidence was missing or the system failed.

Start with real user questions. Evaluate retrieval quality separately from generation. Test the complete workflow, automate repeatable checks and keep human review where mistakes matter. Then feed production failures back into the test set.

The goal is to discover the embarrassing answers before your customers do.

If you want to build, test, deploy and trace RAG agents and workflows with those failures visible, take a look at Fetch Hive.