Your RAG chatbot answered 50 questions perfectly in your demo to the founders at Razorpay. Everyone claps. You ship it. Three weeks later, a customer asks about refund timelines and the bot confidently states a policy that does not exist. Nobody caught it because nobody was testing for it. The bot sounded fluent, so everyone assumed it was correct. This is the core problem with AI systems: they fail silently. A traditional function either returns the right value or throws an error. An LLM returns a confident, well formatted, grammatically perfect answer whether it is right or completely wrong. You cannot tell the difference by reading the output. You need a system that checks for you. **Evaluation** is the practice of systematically measuring whether an AI system produces correct, useful, and safe outputs. **Testing** is the practice of catching regressions before they reach users. Together they are the difference between "it seemed to work when I tried it" and actually knowing your system works. > 📌 **Remember:** A fluent answer and a correct answer are not the same thing. Evaluation > exists because LLMs are equally confident when right and when wrong. ### Why this comes before production, not after Most teams treat evaluation as something you add after an incident. That is backwards. As covered in the RAG module, retrieval and generation are two separate systems working together, and each can fail independently without the other noticing. If you do not have a way to measure both, you will not know which one broke when a user complains. Building a golden dataset and scoring script costs a few hours. An uncaught regression that ships to production costs a lot more, in user trust, in support tickets, and in engineering time spent reverse engineering what changed. ### Where this fits in the system you are building Every module before this one taught you how to build a piece of the system: retrieval in the RAG module, tool calling in the Agents module, serving in Production. Evaluation is not a separate piece sitting next to those. It is the feedback loop that tells you whether the pieces you built are actually working, and it is what lets you change a prompt or swap a model without breaking things you cannot see. ---
Before testing anything specific to RAG or agents, you need the general vocabulary and methods that apply to any AI system. ### Why you cannot just "eyeball" the output Reading ten outputs and deciding they look fine does not scale, is not repeatable, and is not objective. Two engineers reading the same output will disagree. The same engineer will disagree with themselves on different days. You need a fixed, repeatable process. ### What a golden dataset is A **golden dataset** is a fixed set of test cases, each with an input and a known-correct expected output, that you run your system against every time you make a change. Think of it like a driving test. You do not evaluate a new driver by watching them drive randomly for an hour. You give them a fixed set of scenarios, parallel parking, highway merging, and check each one against a known standard. A golden dataset is that same idea applied to your AI system. ```json [ { "id": "hr-001", "question": "How many paid leave days does a new employee get in their first year?", "expected_answer": "12 paid leave days, prorated from the joining month.", "expected_chunks": ["hr-leave-v3-chunk-17"], "expected_sources": ["hr-leave-policy-v3.pdf"] }, { "id": "hr-002", "question": "Can I carry forward unused leave to next year?", "expected_answer": "Up to 5 unused leave days can be carried forward.", "expected_chunks": ["hr-leave-v3-chunk-22"], "expected_sources": ["hr-leave-policy-v3.pdf"] } ] ``` > **Note:** `expected_chunks` records the exact chunk that should have been retrieved to > answer correctly, and `expected_sources` records the document it came from. You need > the chunk-level detail for accurate retrieval scoring later in this module, not just > the document name, since a document can appear in your results while the specific > chunk that actually contains the answer is still missing. Twenty to thirty well chosen test cases, covering common questions, edge cases, and known tricky scenarios, is enough to build your first evaluation suite. That is a starting point, not a finished one. Production systems should keep expanding this dataset over time, adding real failures pulled from production traffic and representative questions users actually asked, not just the cases you thought of on day one. ### LLM-as-judge and where it breaks down Manually checking every output against your golden dataset does not scale either. This is where **LLM-as-judge** comes in: using a separate LLM call to grade whether an output matches the expected answer, instead of a human doing it by hand every time. ```python def judge_answer(question, expected_answer, actual_answer): """ Uses an LLM to score whether actual_answer correctly answers the question, compared against expected_answer. Returns a dict with a score, a verdict category, and a concise reason, parsed from structured JSON output. """ # Asking for structured JSON, not free-form reasoning, keeps the output # easy to validate, store, and compare consistently across eval runs judge_prompt = f""" Question: {question} Expected answer: {expected_answer} Actual answer: {actual_answer} Return JSON with these fields: - score: integer from 1 to 5 - verdict: "correct", "partially_correct", or "incorrect" - reason: one concise sentence explaining the verdict """ response = llm_client.generate(judge_prompt, temperature=0) return json.loads(response) ``` > **Note:** `temperature=0` reduces sampling variability, it does not guarantee the judge > returns the exact same output every time. Across different model versions or provider > updates, the same input can still produce a different score. Pin the judge model to a > specific version where possible, and periodically check that its scores stay consistent > over time rather than assuming determinism is guaranteed by the temperature setting alone. LLM-as-judge is fast and cheap compared to human review, but it has real limits. A judge model can be fooled by confident sounding wrong answers, can have its own biases toward longer or more formal responses, and its scores drift when you switch judge models. Never treat LLM-as-judge as infallible. Spot check its scores against human judgment periodically, especially early on, to confirm it agrees with what a human would actually say. > 🔴 **Common Mistake:** Trusting LLM-as-judge scores without ever sanity checking them > against a human. Teams do this because manual review feels slow, then discover months > later the judge had been silently passing bad answers because they sounded confident. > The fix: spot check 10 to 15 percent of judged outputs by hand every few weeks, and > whenever you switch judge models. ### Human evaluation and pairwise comparison For a smaller set of especially important or ambiguous cases, human evaluation is still the gold standard. **Pairwise comparison** is often more reliable than asking a human to score an answer 1 to 5 in isolation: show the human two candidate answers side by side and ask which is better. People are much more consistent at comparing two things than at assigning an absolute score to one thing. ### Choosing task-specific metrics over generic scores A generic score like "overall quality: 7/10" tells you almost nothing actionable. It does not tell you whether the problem was retrieval, generation, tone, or factual accuracy. Always prefer metrics that map to a specific thing you can fix. This principle carries through everything else in this module. > 📌 **Engineering Decision:** Build a golden dataset and scoring script before shipping > to real users, not after an incident. Twenty to thirty good test cases cost little > upfront. An uncaught regression discovered by an angry customer costs far more. ---
This is the section every AI engineer building a RAG system needs to internalize completely: a RAG system can fail in two completely independent ways, and if you only measure the final answer, you cannot tell which one happened. ### Why splitting the two questions matters Picture two failure scenarios. In the first, your retriever pulls back three completely irrelevant chunks about the wrong policy document, and the model still generates a plausible-sounding but wrong answer because that is what language models do when given bad context. In the second, your retriever does its job perfectly and pulls exactly the right chunk, but the model ignores it and answers from its own outdated training data anyway. Both scenarios produce the same symptom: a wrong answer. But the fix is completely different. The first needs better retrieval. The second needs a stronger instruction to the model to only use the provided context. If you only measure "was the final answer correct," you cannot tell these two apart, and you will waste time fixing the wrong thing. Question 1: Did we retrieve Question 2: Did the model use the right information? that information correctly? | | v v +----------------+ +--------------------+ | Retrieval | | Generation | | metrics | | metrics | | recall@k | | faithfulness | | precision@k | | answer relevance | | MRR | | | | context recall | | | +----------------+ +--------------------+ This diagram sits directly under the two-question framing above, so its purpose is already clear before it appears: everything on the left checks the retriever, everything on the right checks the generator, and a wrong final answer needs to be traced to one side or the other before you can fix it. ### Retrieval-ranking metrics: did we find the right information These metrics only look at what got retrieved, they do not care what the model did with it afterward. All three are scored at the chunk level, against the `expected_chunks` field in your golden dataset, not just whether the right document showed up somewhere. **Recall@k** measures whether the correct chunk appears anywhere in the top k retrieved results. ```python def recall_at_k(retrieved_chunk_ids, expected_chunk_ids, k): """ Returns 1 if any expected chunk appears in the top k retrieved chunk ids, otherwise 0. Scored per question, then averaged across the golden dataset to get an overall recall@k for the system. """ top_k = set(retrieved_chunk_ids[:k]) expected = set(expected_chunk_ids) return 1 if top_k & expected else 0 ``` **Precision@k** measures what fraction of the top k retrieved chunks are actually relevant. High recall with low precision means you are retrieving the right chunk, but burying it under a pile of irrelevant ones, which as covered in the RAG module, drowns the signal the generation step needs. ```python def precision_at_k(retrieved_chunk_ids, relevant_chunk_ids, k): """ Returns the fraction of the top k retrieved chunks that are actually relevant. relevant_chunk_ids can include expected_chunks plus any other chunks a human has separately marked as acceptable for this question. """ retrieved = retrieved_chunk_ids[:k] relevant = set(relevant_chunk_ids) if not retrieved: return 0.0 hits = sum(chunk_id in relevant for chunk_id in retrieved) return hits / len(retrieved) ``` **MRR (Mean Reciprocal Rank)** measures how high up the correct result ranks, not just whether it appears. If the correct chunk is the first result, that scores 1.0. If it is the third result, that scores 1/3. MRR rewards systems that put the right answer at the top, not just somewhere in the list. ```python def reciprocal_rank(retrieved_chunk_ids, correct_chunk_id): """ Returns 1/rank of the correct chunk in the retrieved list, or 0 if it is not present at all. Rank is 1-indexed, so a correct chunk at position 1 scores 1.0, at position 2 scores 0.5, and so on. """ for rank, chunk_id in enumerate(retrieved_chunk_ids, start=1): if chunk_id == correct_chunk_id: return 1 / rank return 0 # correct chunk never showed up in the retrieved results def mean_reciprocal_rank(all_results): """ all_results is a list of (retrieved_chunk_ids, correct_chunk_id) pairs, one per question in the golden dataset. Averages the reciprocal rank across every question to get a single MRR score for the whole system. """ scores = [reciprocal_rank(r, c) for r, c in all_results] return sum(scores) / len(scores) ``` **Context recall** is a related but distinct idea worth not confusing with recall@k. Recall@k is scored against known relevant items, chunks or documents you already labelled in your golden dataset. Context recall instead asks a broader question: does the full retrieved context, taken together, actually contain the information needed to answer the question, even if that information is spread across multiple chunks rather than living in one single labelled chunk. Context precision extends the same idea to the generation side, checking how much of the retrieved context was actually necessary versus noise. ### Generation-quality metrics: did the model use it correctly These metrics assume retrieval already happened and only look at what the model did with the context it was given. Every function below expects `retrieved_context` as a single formatted string, built by joining the text of every retrieved chunk together, not the raw list of retrieval objects your retriever returns. ```python def format_context(retrieved_chunks): """ Converts a list of retrieved chunk objects into a single string of context text, the format every judge function below expects. Call this once after retrieval, before passing context into generation or scoring. """ return "\n\n".join(chunk["text"] for chunk in retrieved_chunks) ``` **Faithfulness (also called groundedness)** measures whether the generated answer is actually supported by the retrieved context, or whether the model added claims that are not in the source material at all. This is the direct check against hallucination. ```python def score_faithfulness(question, retrieved_context, generated_answer): """ Uses an LLM judge to check whether every claim in generated_answer is supported by retrieved_context. Returns a score from 0 to 1, where 1 means fully grounded and 0 means entirely unsupported. """ judge_prompt = f""" Context: {retrieved_context} Answer: {generated_answer} Return JSON with: - unsupported_claims: list of claims in the answer not backed by the context - faithfulness_score: float from 0 to 1 """ response = llm_client.generate(judge_prompt, temperature=0) return json.loads(response)["faithfulness_score"] ``` **Answer relevance** measures whether the answer actually addresses the question that was asked, independent of whether it is factually grounded. A perfectly faithful answer that does not address the question is still a failure. ```python def score_relevance(question, generated_answer): """ Uses an LLM judge to check whether generated_answer actually addresses question, regardless of whether it is factually correct. Returns a score from 0 to 1, where 1 means fully on-topic and directly responsive. """ judge_prompt = f""" Question: {question} Answer: {generated_answer} Does the answer directly address what was asked, regardless of whether it is factually correct? Return JSON with: - relevance_score: float from 0 to 1 - reason: one concise sentence """ response = llm_client.generate(judge_prompt, temperature=0) return json.loads(response)["relevance_score"] ``` > **Note:** Asking the judge to return structured JSON with a specific evidence field, > `unsupported_claims` for faithfulness, rather than just a bare number, produces more > reliable scores and is easier to validate and store than free-form reasoning text. It > also forces the judge to point at something concrete instead of just outputting a > number that pattern matches on overall fluency. > 🔴 **Common Mistake:** Evaluating only the final answer's "helpfulness" without ever > checking faithfulness separately. A confident, well written, completely fabricated > answer scores just as high as a correct one on a generic helpfulness metric. The fix: > always score faithfulness and relevance as two separate numbers, never one blended score. | Metric | Question it answers | Failure it catches | |:---|:---|:---| | Recall@k | Was the correct chunk retrieved at all? | Missing the correct chunk entirely | | Precision@k | How much noise is in the retrieved set? | Correct chunk buried under irrelevant ones | | MRR | How high does the correct chunk rank? | Correct chunk present but ranked low | | Faithfulness | Is the answer grounded in retrieved context? | Hallucination despite good retrieval | | Answer relevance | Does the answer address the actual question? | On-topic context, off-topic answer | ### Debugging retrieval failures using this split When a golden dataset test case fails, use the two-question split to diagnose it instead of guessing: 1. Check recall@k first. If the expected chunk was never retrieved, the problem is in retrieval, chunk size, embedding model mismatch, or a missing metadata filter, not generation. 2. If recall@k is fine but faithfulness is low, the retriever did its job and the problem is in generation, likely the prompt is not instructing the model strongly enough to stick to the provided context. 3. If both are fine but the answer still scored poorly, check answer relevance, the model may be technically grounded but answering a slightly different question than the one asked. ---
Agent evaluation is harder than RAG evaluation because an agent takes a sequence of actions, and the final result can look correct even when the path to get there was wasteful, risky, or got lucky. ### Task completion rate The most basic agent metric: out of a set of test scenarios, what fraction did the agent successfully complete end to end. This is necessary but not sufficient on its own, a high completion rate can hide serious inefficiency. ### Tool selection accuracy For each step in a scenario, did the agent choose an acceptable tool for that situation. As covered in the Agents module, overlapping or poorly described tools cause the agent to pick the wrong one. This metric catches that directly, separate from whether the overall task eventually succeeded anyway through a roundabout path. ### Trajectory evaluation **Trajectory evaluation** looks at the entire sequence of steps an agent took, not just whether it arrived at the right destination. Two agents can both successfully process a PhonePe-style refund request, but one does it in 3 clean steps while the other takes 11 steps, calls the wrong tool twice, and only succeeds because it eventually stumbled onto the right path. > 📌 **Remember:** A successful outcome with a messy trajectory is still a reliability > risk. The next time that same messy path is taken, it might not stumble into success. Agents rarely follow one single correct path. A search-verify-update sequence and a search-search-verify-update sequence can both be legitimate, the second is just less efficient. Do not evaluate an agent's trajectory against one fixed expected sequence, that is too rigid and will flag valid alternate paths as failures. Instead, evaluate it against a set of constraints: which tools were allowed, which tools should never have been called for this scenario, which actions were required to happen at all, and whether required actions happened in a valid order. ```python def evaluate_trajectory(actual_steps, allowed_tools, forbidden_tools, required_tools): """ Scores an agent's trajectory against constraints rather than one fixed expected sequence. allowed_tools is the full set of tools that were valid to use anywhere in this scenario. forbidden_tools should never have been called. required_tools must appear at least once, in any order, for the task to count as correctly completed. """ used_tools = [step["tool"] for step in actual_steps] forbidden_calls = [t for t in used_tools if t in forbidden_tools] disallowed_calls = [t for t in used_tools if t not in allowed_tools] missing_required = [t for t in required_tools if t not in used_tools] return { "total_calls": len(used_tools), "forbidden_calls": forbidden_calls, "disallowed_calls": disallowed_calls, "missing_required_tools": missing_required, "constraint_violation": bool(forbidden_calls or disallowed_calls or missing_required) } ``` > **Note:** This does not try to declare one exact path as correct. It checks that the > agent stayed inside its allowed toolset, never touched a forbidden tool, and eventually > called everything required. A longer path that satisfies all three constraints still > passes, it just costs more, which is exactly what the cost-per-task metric below is for. ### Failure rate and unnecessary tool calls Track how often the agent fails outright, and separately, how many tool calls succeeded but were not actually needed to complete the task. Unnecessary calls are not free, they cost latency and money on every single run, even the ones that technically succeed. ### Cost per successful task The metric that ties everything together for a business stakeholder. As an illustrative example only, since real cost depends heavily on token usage, model choice, retries, and which external APIs get called, an agent with a 95 percent completion rate that costs ₹40 per task might be worse for the business than an agent with an 85 percent completion rate that costs ₹4 per task, depending on the use case. Always report cost alongside completion rate, never completion rate alone. | Metric | What it reveals | |:---|:---| | Task completion rate | Did the agent finish the job at all | | Tool selection accuracy | Did it pick an acceptable tool at each step | | Trajectory evaluation | Did it stay inside its constraints, or violate them | | Failure rate | How often does it fail outright | | Cost per successful task | Is a high success rate actually affordable | A RAG system retrieves 10 semantically similar chunks for a user's question, but the chunk containing the correct answer consistently lands at position 7, well below where most generation prompts pay close attention. Recall@10 looks fine since the correct chunk is technically present in the retrieved set, so recall alone will not surface this problem. What will surface it is MRR, since a chunk ranked seventh scores only 1/7 on reciprocal rank even though it counted as a hit for recall. The fix that most directly addresses this exact symptom is a reranking step applied after initial retrieval: it reorders an already-retrieved candidate set by relevance so the best chunk moves toward the top, directly improving MRR without needing to change what gets retrieved in the first place. ---
Evaluation before launch catches known problems. Production evaluation catches the problems you did not think to test for, and testing infrastructure prevents you from reintroducing problems you already fixed. ### Latency, cost, success rate, and user feedback as ground truth Once a system is live, real user behavior becomes a data source your golden dataset cannot replicate. Track latency and cost per request as baseline health signals. Track explicit user feedback, thumbs up or down, and implicit signals like whether a user immediately rephrased their question, which usually means the first answer did not help. ### Drift and regression are detected differently **Drift** is when your system's real-world performance degrades over time without any code change, usually because the type of questions users ask has shifted, or the underlying data your RAG system indexes has gone stale. A **regression** is a performance drop caused directly by a change you made, a new prompt, a new model version, an updated chunking strategy. The two are caught by different processes, and understanding why is what makes offline evaluation and production monitoring complementary rather than redundant. A regression is caught by re-running the same golden dataset and the same evaluation setup against a new system version, and comparing the score against a saved baseline before that version ships. Drift is caught by watching production traffic itself, real user queries, changing data distribution, a retrieval corpus that has gone stale, none of which a fixed golden dataset run before deployment can see coming. The way to catch a regression before it reaches users is to run your golden dataset against every proposed change before merging it, not after deploying it. > 🔴 **Common Mistake:** No regression tests in place, so a prompt change that fixes one > reported issue silently breaks three other cases that were previously working. Nobody > notices until a different customer complains about a different question. The fix: treat > your golden dataset as a required check in your deployment pipeline, the same way you > would treat a unit test suite for regular code. ### The testing pyramid for AI systems Different test types catch different classes of problems, and none of them alone is enough. * **Unit tests** check individual functions, does your chunking function split text at the correct boundaries, does your JSON parser correctly handle a malformed response. These are fast, deterministic, and should run on every commit. * **Integration tests** check that components work together, does a query actually retrieve from the vector database and produce a generated answer end to end, using a small fixed test index rather than production data. * **Prompt regression tests** run your golden dataset against the current prompt and compare scores against a saved baseline, catching the exact scenario described above. * **Adversarial tests** deliberately try to break the system, malformed inputs, prompt injection attempts, edge case questions designed to confuse retrieval. As covered in the AI Safety module, this overlaps directly with security testing. * **Synthetic test generation** uses an LLM to generate new plausible test cases at scale, useful for expanding coverage beyond what you can hand write, though synthetic cases should always be spot checked by a human before being trusted as ground truth. ```bash ## Run the full evaluation suite before merging a prompt or retrieval change ## This script scores the golden dataset and compares against the saved baseline python run_eval.py --dataset golden_hr_qa.json --compare-baseline ## Output shows a score drop, this is what should block the merge ## faithfulness: 0.91 -> 0.74 (REGRESSION DETECTED) ## answer_relevance: 0.88 -> 0.89 (stable) ``` > **Note:** `--compare-baseline` is a flag telling the script to load the previously > saved scores and diff against the current run, rather than just printing the current > scores in isolation. Without a baseline to compare against, a score has no context. ### Release gates: deciding what blocks a deploy A score comparison alone does not tell you whether to ship. "Did faithfulness go up or down slightly" is the wrong question to gate a release on. The right approach is to define explicit minimum thresholds upfront, before you need them, so a deploy decision becomes a checklist rather than a judgment call made under pressure. Golden dataset | v Evaluate | v Compare baseline | v Mandatory metrics above threshold? | ---------------- | | YES NO | | v v Merge/deploy Block + investigate A release gate is a small set of hard minimums the system must clear, for example: ```text Recall@5 >= 0.90 Faithfulness >= 0.95 Critical cases = 100% pass Cost per task <= your budget ceiling p95 latency <= your latency ceiling ``` > **Note:** These exact numbers are illustrative examples, not universal standards. The > right thresholds depend on your specific use case, how costly a wrong answer is for > your users, and your actual latency and cost budget. Set them deliberately for your > system rather than copying numbers from elsewhere. "Critical cases" deserve special treatment inside a golden dataset: a handful of test cases where getting the answer wrong is unacceptable regardless of overall average score, a wrong refund policy answer or a wrong safety instruction, for instance. Require 100 percent pass on that subset specifically, separate from the general average across the rest of the dataset. > 📌 **Engineering Decision:** Define mandatory minimum thresholds for your critical > metrics before you need them, not while an incident is happening. A release gate turns > "does this look okay" into a checklist an automated pipeline can enforce consistently. ### Evaluation set leakage One more risk worth naming explicitly for an advanced system: if you repeatedly tune prompts, retrieval settings, or model choices against the exact same golden dataset, you can end up optimizing for those specific questions without the system actually generalizing to real user traffic. This is the same overfitting problem you would guard against with a held-out test set in any other machine learning workflow, and the fix is the same idea. Keep a development set you iterate against freely, and a separate held-out set that is never used for tuning, only for a final check before a release. Add new production failures into the development set as you discover them, and periodically refresh the held-out set so it does not go stale either. ---
> **Note:** These commands assume a Linux or macOS shell, or WSL on Windows. On native > Windows without WSL, replace `touch` with a text editor or `New-Item` in PowerShell. 1. Create a golden dataset of 20 question-answer pairs for the RAG chatbot from the RAG module, covering common questions, one ambiguous edge case, and one question the documents genuinely do not answer. Include `expected_chunks` for each entry. ```bash ## Create the project folder and the golden dataset file mkdir rag-eval-lab && cd rag-eval-lab touch golden_hr_qa.json ## fill this with 20 entries following the schema shown earlier ``` 2. Install dependencies and implement the scoring functions defined earlier in this module: `format_context`, `score_faithfulness`, and `score_relevance`. ```bash ## Install the packages needed for the eval harness pip install openai pandas ## pandas is used to tabulate and summarise the scores ``` 3. Write the evaluation harness, making the data contract between retrieval and scoring explicit rather than assumed. ```python ## eval_harness.py def run_evaluation(golden_dataset, rag_system): """ Runs every question through the RAG system, then scores retrieval and generation separately using the functions defined earlier in this module. retrieved_context is built explicitly via format_context so the contract between retrieval output and the judge functions is clear. """ results = [] for case in golden_dataset: retrieved_chunks = rag_system.retrieve(case["question"]) retrieved_context = format_context(retrieved_chunks) answer = rag_system.generate(case["question"], retrieved_context) retrieved_ids = [c["chunk_id"] for c in retrieved_chunks] results.append({ "id": case["id"], "recall_at_5": recall_at_k(retrieved_ids, case["expected_chunks"], k=5), "precision_at_5": precision_at_k(retrieved_ids, case["expected_chunks"], k=5), "faithfulness": score_faithfulness(case["question"], retrieved_context, answer), "relevance": score_relevance(case["question"], answer) }) return results ``` 4. Run the harness, then deliberately look at any case where recall@5 succeeded but faithfulness scored low, and any case where recall@5 failed. Confirm you can tell which failure is which from the numbers alone. ```bash ## Run the harness and print a summary table split by failure type python eval_harness.py --dataset golden_hr_qa.json --report-by-failure-type ``` 5. Build an evaluation loop for the agent from the Agents module, scoring task completion rate, `evaluate_trajectory` against allowed/forbidden/required tool constraints, and cost per successful task across 15 scenarios. 6. Write a prompt regression test: save the current scores as a baseline, deliberately change one word in the system prompt to make it worse, rerun the evaluation, and confirm the script flags the regression. ```bash ## Save the current scores as the baseline to compare future runs against python eval_harness.py --dataset golden_hr_qa.json --save-baseline ## Expected output showing the harness correctly caught the deliberate regression ## faithfulness: 0.91 -> 0.68 (REGRESSION DETECTED) ## Test FAILED - do not merge this prompt change ``` 7. Define a release gate for this system using the threshold format shown earlier, and confirm the deliberately broken prompt from step 6 fails that gate. Success looks like: your script reliably flags the deliberate regression in step 6, your release gate correctly blocks that same broken version in step 7, and in step 4 you can point to specific numbers, not a gut feeling, that tell you whether a given failure was a retrieval problem or a generation problem. ---
Your RAG chatbot answered 50 questions perfectly in your demo to the founders at Razorpay. Everyone claps. You ship it. ...
Before testing anything specific to RAG or agents, you need the general vocabulary and methods that apply to any AI syst...
This is the section every AI engineer building a RAG system needs to internalize completely: a RAG system can fail in tw...
Agent evaluation is harder than RAG evaluation because an agent takes a sequence of actions, and the final result can lo...
Evaluation before launch catches known problems. Production evaluation catches the problems you did not think to test fo...
> Note: These commands assume a Linux or macOS shell, or WSL on Windows. On native > Windows without WSL, replace touch ...
Concept What it measures When to use it Golden dataset Fixed test cases with known-correct answers Foundation for every ...
[image-1] Split-question RAG evaluation diagram: [Style: Clean flat diagram on white background] -> [Title: "Two Questio...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.