Harness Engineering: Building the Test Infrastructure Your AI Agents Actually Need

The Problem Nobody Wants to Admit

Traditional software has a well-understood testing contract: given a specific input, the correct output is fixed and known. Write a test, assert a value, ship with confidence. AI systems break this contract completely.

Ask an LLM the same question twice and you get two different — possibly equally valid — answers. Ask it in slightly different words and the answer changes again. Swap the underlying model version and everything shifts. Add a new document to your retrieval index and old answers that were correct become wrong.

This non-determinism is not a bug. It is the feature that makes AI systems useful. But it creates a testing gap that most engineering teams quietly ignore until something fails in production — a hallucinated fact, a biased response, a confidently wrong answer to a compliance question.

Harness engineering is the answer. It is the systematic practice of building evaluation infrastructure — test harnesses — that can measure AI system quality despite the absence of fixed correct outputs. Think of it as software quality engineering, re-invented for probabilistic systems.

What Is a Test Harness, Exactly?

In classical software engineering, a test harness is the scaffolding that surrounds a component under test: it provides inputs, captures outputs, and verifies results. For an AI system, the same concept applies — but the components are richer and the verification layer is far more complex.

An AI test harness has five major components working together:

Why AI Systems Can't Be Tested Like Regular Code

The core challenge is that AI outputs exist on a quality spectrum, not a pass/fail binary. Consider these two answers to the question "What is the capital of France?":

• "Paris is the capital of France." → Correct.
• "The capital of France is Paris, located on the Seine river, with a population of roughly 2 million in the city proper." → Also correct, and more informative.

Both answers pass a naive exact-match test — or both fail if the expected string is just "Paris". Neither approach is useful. You need to measure correctness, completeness, groundedness, and relevance as separate, graded dimensions.

Now scale this to a multi-agent system where one agent retrieves documents, another summarises them, a third generates a response, and a fourth validates it for compliance. The failure surface is enormous. A tiny prompt change in agent one can cascade into a completely different final answer ten steps later.

This is why harness engineering matters: you cannot reason about system quality without a measurement infrastructure that matches the system's complexity.

The Three Levels of AI Evaluation

Borrowed from software testing strategy, evaluation in AI systems runs at three levels. You need all three — skipping any one creates dangerous blind spots.

The LLM-as-Judge Pattern

One of the most important advances in AI evaluation is using a powerful LLM to evaluate the outputs of another LLM. This is called the LLM-as-Judge pattern, and it is now the de-facto approach for evaluating free-text generation at scale.

Here is how it works: you provide the judge LLM with a structured rubric, the original question, optionally the retrieved context, and the model's answer. The judge scores the answer on each dimension and returns a structured score with reasoning.

For example, to evaluate faithfulness (does the answer stay grounded in the source documents?), the judge prompt looks roughly like:

"Given the following context: [CONTEXT]. Evaluate whether the answer: [ANSWER] contains any claims not supported by the context. Return a score from 0 to 1 where 1 = fully faithful, and explain each unsupported claim."

Research from frameworks like RAGAS and DeepEval shows that GPT-4-class models acting as judges achieve strong correlation with human evaluator scores — typically above 0.85 Pearson correlation — making this a viable alternative to expensive manual annotation.

Important caveats: LLM judges can be biased toward their own outputs, verbose answers, and positional preferences (preferring the first response when comparing two). Mitigate this by using a judge model from a different family than your application model, randomising the order of responses, and calibrating judge outputs against human labels on a small reference set.

Building Your Golden Dataset

The most important artefact in your harness is the golden dataset — a curated collection of representative test cases. Building it is not glamorous, but it is the foundation everything else stands on.

A good golden dataset covers:

• Happy path cases — typical questions the system should handle confidently.
• Edge cases — ambiguous queries, multi-hop reasoning requirements, cross-domain questions.
• Adversarial cases — jailbreak attempts, prompt injection, leading questions designed to elicit hallucinations.
• Regression anchors — cases that previously failed and were fixed, ensuring you never re-introduce a known bug.
• Distribution samples — cases drawn from real production traffic, sampled periodically and labelled by domain experts.

A practical starting point for most teams is 200–500 curated cases. This is enough to detect significant regressions while remaining manageable to maintain. Expand it continuously — every production failure should become a new test case.

The Key Evaluation Metrics

Different types of AI systems need different metrics. Here are the most important ones for RAG-based and agentic systems:

• Faithfulness — Do generated claims stay grounded in the retrieved context? The primary anti-hallucination metric.
• Answer Relevancy — Does the response actually address what the user asked? High faithfulness with low relevancy = correct but useless.
• Context Precision — Of the documents retrieved, what proportion were actually useful? Measures retrieval signal-to-noise ratio.
• Context Recall — Did retrieval surface all the documents needed to answer correctly? Measures retrieval completeness.
• Hallucination Rate — The percentage of responses containing at least one factual claim not supported by context.
• Tool Call Accuracy — For agents: did the agent call the right tools, with the right arguments, in the right order?
• Task Completion Rate — For multi-step agents: what percentage of tasks are completed end-to-end without human intervention?
• Latency P95 — The 95th percentile end-to-end response time. AI systems that are accurate but slow fail in production.

Wiring Evaluation Into CI/CD

The most operationally mature teams treat evaluation as a deployment gate — not a quarterly review. Every pull request that changes a prompt, model version, retrieval strategy, or agent configuration triggers a full evaluation run before merging.

The typical CI/CD harness pipeline looks like this:

1. Trigger — PR opened or commit pushed to main.
2. Run — Evaluation framework executes all test cases against the candidate build.
3. Score — LLM judge scores outputs; metrics are aggregated.
4. Compare — Scores are compared to the baseline (last passing build).
5. Gate — If any metric regresses beyond a defined threshold, the pipeline fails and the PR is blocked.
6. Report — A structured report is posted to the PR with per-case breakdowns, so the engineer can see exactly which cases failed and why.

This approach shifts AI quality assurance left — catching regressions at the point of change rather than in production. The cost is a slightly longer CI run (typically 5–15 minutes for 200-case suites). The benefit is that you never ship a model change that silently breaks 10% of your user scenarios.

Evaluation Frameworks: A Practical Comparison

You do not need to build your evaluation infrastructure from scratch. Mature open-source and commercial frameworks exist. Here is how the major ones compare:

Common Mistakes That Undermine Your Harness

Even teams that invest in evaluation infrastructure often make a handful of recurring mistakes that limit its value:

• Testing only the happy path. Golden datasets full of straightforward questions will produce optimistic scores. The cases that matter most — ambiguous inputs, multi-hop reasoning, adversarial prompts — are the ones most teams skip because they are harder to label.
• Using the same LLM as builder and judge. A model evaluated by a judge from the same family tends to receive inflated scores. Use a different model family for evaluation — if you build with Claude, judge with GPT-4o.
• Setting thresholds without baselines. A faithfulness score of 0.82 means nothing without knowing whether your previous version scored 0.85 or 0.60. Always run evaluation on the current production version first to establish a baseline before setting gate thresholds.
• Treating evaluation as a one-time activity. Models drift, retrieval indices change, user behaviour evolves. Evaluation is not a launch checklist item — it is a continuous operational discipline.
• Ignoring latency. Accuracy metrics capture whether answers are correct. Latency captures whether users will actually wait for them. A system with 0.95 faithfulness but 12-second P95 latency is not production-ready.

What a Mature Harness Looks Like in Practice

A production-grade AI test harness at a mature organisation typically includes:

• A versioned golden dataset of 500+ cases, maintained in a Git repository with change history.
• Automated nightly evaluation runs covering all registered model versions and retrieval configurations.
• A CI/CD gate that blocks any model or prompt change that causes more than a 2% regression on core metrics.
• A human annotation queue where production samples flagged as low-confidence are sent for expert labelling, then added to the golden dataset.
• A monitoring dashboard tracking live evaluation metrics sampled from production traffic — separate from but complementary to the offline harness.
• A documented incident protocol: when evaluation catches a regression, there is a defined playbook for rollback, root-cause analysis, and test case enrichment.

Most teams are not here yet — and that is fine. The point is to start. Even a 50-case golden dataset and a single faithfulness metric wired into CI is infinitely better than shipping blind.

Where to Start

If you are building your harness from zero, the right sequence is:

1. Identify the three to five most critical user scenarios your system must handle correctly — these become your first golden cases.
2. Pick one evaluation framework (DeepEval if you want pytest integration; RAGAS if your system is retrieval-heavy; Promptfoo if you are mainly testing prompts).
3. Define your minimum acceptable thresholds for faithfulness and answer relevancy — these are your first deployment gates.
4. Run evaluation on your current production system to establish a baseline.
5. Wire evaluation into your CI pipeline. Any change to prompt, model, or retrieval configuration must pass before it ships.
6. Treat every production failure as a new test case — add it to the golden dataset within 48 hours of root-cause analysis.

Harness engineering is not a project you complete. It is a capability you build incrementally — and every iteration makes your AI system safer, more predictable, and more trustworthy.

References

1. Shahul Es, Jithin James, Luis Espinosa Anke, Steven Schockaert. "RAGAS: Automated Evaluation of Retrieval Augmented Generation." arXiv:2309.15217, 2023. arxiv.org/abs/2309.15217
2. Confident AI. "DeepEval — The Open-Source LLM Evaluation Framework." Documentation, 2024. docs.confident-ai.com
3. Zheng, L., Chiang, W., Sheng, Y., et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023. arxiv.org/abs/2306.05685
4. Promptfoo. "Test and Evaluate LLMs — Open Source LLM Testing." Documentation, 2024. promptfoo.dev/docs/intro