Build a Deterministic AI Test Harness Without Calling a Real LLM

Your CI pipeline should not need a model provider’s uptime, pricing, latency, or mood to pass.

The first time this hits you, it usually looks innocent. A test asserts that an assistant replies with "Your refund has been processed." The model sometimes returns "I’ve processed your refund." Sometimes it adds a polite sentence. Sometimes it returns the same text with different whitespace. The test becomes flaky, so someone adds .strip(), then someone else adds a regex, then someone suggests “just call the real model five times and take a majority vote.” At that point, your test suite has become a small, expensive science experiment.

A deterministic AI test harness solves a different problem: it lets you test your system’s behavior around the model without depending on the model itself. You can test prompt assembly, routing, tool selection, schema extraction, retry logic, guardrails, state transitions, and failure handling — all without making a single network call to a real LLM.

This matters even more in 2026 because many production AI features are not “send a prompt, get text.” They are loops: read context, call tools, validate output, retry on malformed responses, escalate when uncertain. The interesting bugs are rarely “the model was not smart enough.” They are usually:

• the agent called the wrong tool,
• the tool arguments were subtly wrong,
• the parser accepted invalid JSON,
• the retry loop retried forever,
• the prompt changed unexpectedly,
• the response violated a policy rule,
• the state machine skipped a required step.

Those are software bugs. You can test them deterministically.

TL;DR

• Treat the LLM as an external dependency, like a payment gateway or database.
• Put all model access behind a narrow interface you control.
• Use a scripted fake LLM for unit and integration tests.
• Assert structure, tool traces, schemas, and policies — not exact prose.
• Use replayed production traffic for regression tests after scrubbing.
• Keep real LLM evaluation in a separate nightly or canary pipeline.

---

📋 Table of Contents

• The Real Problem
• 1. Decide What Must Be Deterministic
• 2. Hide the Model Behind a Narrow Port
• 3. Script the Model With a Fake LLM
• 4. Normalize Outputs Before You Assert Anything
• 5. Test the Tool-Call Loop, Not Just the Final Answer
• 6. Simulate the Model's Bad Days
• 7. Replay Production Traffic Without Paying for It
• 8. Add Deterministic Evaluators That Do Not Need a Model
• 9. Put Real LLMs in a Nightly Canary, Not CI
• What I Would Actually Ship

---

The Real Problem

The mistake most teams make is treating the LLM as the system under test when they should be testing the application around the LLM.

If you build a support assistant, your application probably does several things before and after the model is involved:

1. It gathers user context.
2. It builds a prompt.
3. It chooses available tools.
4. It sends a request to a model.
5. It receives text, structured data, or tool calls.
6. It executes tools.
7. It validates results.
8. It decides whether to answer, retry, or escalate.

Only one of those steps actually requires a real model. The rest are ordinary code. If your tests require a live LLM for all of them, you are paying money and adding latency to test code that could otherwise be fast, repeatable, and precise.

A deterministic AI test harness does not pretend the model is perfect. It replaces the model with a controlled test double so the rest of the system can be verified.

That distinction is the whole game.

---

1. Decide What Must Be Deterministic

Scenario: Your test asks an assistant, “Where is my order?” The real model sometimes says, “Your order shipped yesterday,” and sometimes says, “It looks like your order shipped yesterday.” Your test fails because it expected an exact string.

Why it matters: Exact prose assertions are usually testing the model’s style, not your product behavior. Unless you are rendering a legally required disclosure or a fixed UI string, natural-language variation is not the thing you want your test suite to police.

Before writing the harness, split your AI feature into layers:

Layer	Example	Deterministic?	What to test
Prompt construction	System prompt + customer context	Yes	Snapshot or structural assertions
Tool availability	Only refund tool for billing agents	Yes	Policy tests
Tool selection	Model calls lookup_order	Mostly	Trace assertions with fake LLM
Tool arguments	order_id: "A-123"	Mostly	Schema and value assertions
Structured output	JSON with refund_amount	Yes after parsing	Schema validation
Final prose	“Your refund is complete.”	No	Policy checks, not exact text
Side effects	Database update, ticket created	Yes	State assertions

The goal is not to make the model deterministic. The goal is to make your system’s response to the model deterministic.

For example, if the model says:

Your refund has been processed.

or:

I’ve processed your refund. Let me know if you need anything else.

your application may not care. But if the model fails to call process_refund, or calls it with the wrong amount, that is a real defect.

🔍 Why this matters: The most valuable deterministic AI tests are usually not about language quality. They are about routing, tool use, safety constraints, and state changes.

A useful rule:

• If the output is machine-consumed, make the test strict.
• If the output is human-facing prose, make the test policy-based.
• If the output is a side effect, make the test state-based.

---

2. Hide the Model Behind a Narrow Port

Scenario: Your code calls a provider SDK from three different services. Tests try to monkeypatch the client, but one module imports it differently, another wraps it in a retry helper, and a third passes provider-specific objects around. You cannot replace the model cleanly.

Why it matters: If your application is coupled directly to a provider’s request and response shapes, your fake harness becomes fragile. You end up mocking internals instead of substituting a simple boundary.

Create a small interface that represents what your application needs from a model. Not every provider feature — just the contract your code depends on.

from typing import Any, Protocol, Literal
from dataclasses import dataclass

@dataclass(frozen=True)
class ToolCall:
    id: str
    name: str
    arguments: dict[str, Any]

@dataclass(frozen=True)
class LLMResponse:
    content: str | None = None
    tool_calls: tuple[ToolCall, ...] = ()
    finish_reason: Literal["stop", "tool_calls", "length"] = "stop"

class LLMBackend(Protocol):
    def complete(
        self,
        messages: list[dict[str, Any]],
        tools: list[dict[str, Any]] | None = None,
    ) -> LLMResponse:
        ...

This interface is intentionally boring. It does not know whether the real implementation talks to OpenAI, Anthropic, Bedrock, Azure, a local model, or a fake object in a test.

Your production adapter might look like this conceptually:

class ProductionLLM:
    def __init__(self, client):
        self.client = client

    def complete(self, messages, tools=None):
        raw = self.client.create_completion(
            messages=messages,
            tools=tools,
        )
        return normalize_provider_response(raw)

The important part is normalize_provider_response. Provider responses often contain extra fields, nested objects, and provider-specific tool-call representations. Your application should not have to know about those details.

Why this works: Once the model is behind a port, the deterministic harness becomes trivial. Your tests can pass in ScriptedLLM, ReplayLLM, or FailingLLM without changing application code.

⚠️ Gotcha: Do not let provider-specific response objects leak into your domain code. The moment your business logic depends on a provider’s exact JSON shape, your test harness becomes much harder to keep stable.

---

3. Script the Model With a Fake LLM

Scenario: You want to test an agent that first looks up an order, then answers the user. With a real model, you cannot reliably force the exact sequence. Sometimes it answers too early. Sometimes it asks a clarifying question. Your test becomes probabilistic.

Why it matters: Agent behavior is often a loop. You need to control the loop’s inputs to verify the loop itself.

A scripted fake LLM returns responses in a known order. It can also record every request so you can assert what your application sent.

from collections import deque
from copy import deepcopy
from dataclasses import dataclass, field
from typing import Any

class UnexpectedLLMCall(Exception):
    pass

@dataclass
class ScriptedLLM:
    responses: deque[LLMResponse | Exception]
    calls: list[dict[str, Any]] = field(default_factory=list)

    def complete(
        self,
        messages: list[dict[str, Any]],
        tools: list[dict[str, Any]] | None = None,
    ) -> LLMResponse:
        self.calls.append({
            "messages": deepcopy(messages),
            "tools": deepcopy(tools),
        })

        if not self.responses:
            raise UnexpectedLLMCall(
                "The application made an unexpected LLM call."
            )

        item = self.responses.popleft()

        if isinstance(item, Exception):
            raise item

        return item

Now you can script a two-step interaction:

def script(*responses):
    return ScriptedLLM(deque(responses))

fake = script(
    LLMResponse(
        content=None,
        tool_calls=(
            ToolCall(
                id="call_1",
                name="lookup_order",
                arguments={"order_id": "A-123"},
            ),
        ),
        finish_reason="tool_calls",
    ),
    LLMResponse(
        content="Your order shipped yesterday.",
        finish_reason="stop",
    ),
)

This fake says: “First, the model wants to call lookup_order. Then, after receiving the tool result, it produces the final answer.”

You can now test your application’s behavior without caring whether a real model would have phrased the answer differently.

Why this works: The fake LLM is not trying to simulate intelligence. It simulates a sequence. That is exactly what most integration tests need.

A common mistake is making the fake too smart. Do not build a mini-model. Your fake should be obvious, explicit, and slightly stupid. If the fake becomes complex, you now have two systems to test.

💡 Practical note: Store the recorded calls. They are extremely useful for asserting that your prompt contains the expected context, that tools were passed correctly, and that tool results were fed back into the conversation.

---

4. Normalize Outputs Before You Assert Anything

Scenario: Your agent returns structured JSON. One test run produces:

{"refund_amount": 25.0, "currency": "USD"}

Another produces:

{
  "currency": "USD",
  "refund_amount": 25
}

Your string comparison fails even though the data is semantically identical.

Why it matters: Model outputs often vary in formatting, key order, whitespace, and sometimes field aliases. If you assert against raw text, you will spend your life chasing irrelevant differences.

The fix is to parse and canonicalize before asserting.

For JSON-like output, compare parsed objects:

import json

def canonical_json(value: Any) -> str:
    return json.dumps(
        value,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    )

def test_refund_output_is_correct():
    raw = '{"refund_amount": 25.0, "currency": "USD"}'
    parsed = json.loads(raw)

    assert parsed == {
        "refund_amount": 25.0,
        "currency": "USD",
    }

If you need snapshot-style comparisons, canonicalize first:

def test_canonical_output(snapshot):
    parsed = {
        "currency": "USD",
        "refund_amount": 25.0,
    }

    assert canonical_json(parsed) == snapshot

For dynamic fields, redact them before comparing:

REDACT_KEYS = {
    "id",
    "created_at",
    "request_id",
    "trace_id",
}

def redact(value: Any) -> Any:
    if isinstance(value, dict):
        return {
            k: "[REDACTED]" if k in REDACT_KEYS else redact(v)
            for k, v in value.items()
        }

    if isinstance(value, list):
        return [redact(item) for item in value]

    return value

Then:

def test_order_payload_is_stable():
    payload = {
        "id": "ord_98123",
        "status": "shipped",
        "created_at": "2026-01-14T10:22:00Z",
        "items": [{"sku": "SKU-1", "qty": 2}],
    }

    assert redact(payload) == {
        "id": "[REDACTED]",
        "status": "shipped",
        "created_at": "[REDACTED]",
        "items": [{"sku": "SKU-1", "qty": 2}],
    }

For natural-language responses, avoid exact equality unless the text is fixed product copy. Instead, assert invariants:

def test_answer_mentions_refund_without_exact_text():
    answer = "I’ve processed your refund."

    assert "refund" in answer.lower()
    assert "error" not in answer.lower()
    assert len(answer) < 500

This is weaker than exact equality, but that is often correct. The test should encode the requirement, not the model’s stylistic freedom.

Why this works: Normalization turns irrelevant variation into noise and keeps the test focused on semantic correctness.

🧠 The important part: If your test fails because of punctuation, key order, or a synonym, you are probably testing the wrong layer.

---

5. Test the Tool-Call Loop, Not Just the Final Answer

Scenario: Your assistant gives the right final answer, but only because a downstream service happened to return a useful default. Or worse, it calls a tool twice, mutates state twice, and still says “done.” If you only assert the final message, you miss the actual defect.

Why it matters: In agent systems, the path matters as much as the destination. A correct answer produced by the wrong tool call is still a bug.

Build a small deterministic runner around your model port. This runner is your application’s agent loop, stripped of provider details.

import json
from collections.abc import Callable
from dataclasses import dataclass
from typing import Any

@dataclass
class Tool:
    name: str
    schema: dict[str, Any]
    fn: Callable[..., Any]

def run_agent(
    llm: LLMBackend,
    tools: dict[str, Tool],
    initial_messages: list[dict[str, Any]],
    max_steps: int = 6,
) -> tuple[str | None, list[dict[str, Any]]]:
    messages = list(initial_messages)
    events: list[dict[str, Any]] = []

    for _ in range(max_steps):
        response = llm.complete(
            messages,
            tools=[tool.schema for tool in tools.values()],
        )

        events.append({
            "type": "llm_response",
            "tool_calls": [call.name for call in response.tool_calls],
        })

        if not response.tool_calls:
            return response.content, events

        messages.append({
            "role": "assistant",
            "content": response.content,
            "tool_calls": [
                {
                    "id": call.id,
                    "type": "function",
                    "function": {
                        "name": call.name,
                        "arguments": json.dumps(call.arguments),
                    },
                }
                for call in response.tool_calls
            ],
        })

        for call in response.tool_calls:
            tool = tools.get(call.name)

            if tool is None:
                result = {
                    "error": "unknown_tool",
                    "tool": call.name,
                }
            else:
                try:
                    result = tool.fn(**call.arguments)
                except Exception as exc:
                    result = {"error": str(exc)}

            events.append({
                "type": "tool_result",
                "name": call.name,
                "arguments": call.arguments,
                "result": result,
            })

            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result),
            })

    raise RuntimeError("Agent did not finish within max_steps")

Now you can write a test that verifies the full trace.

def test_agent_uses_order_lookup_before_answering():
    fake = script(
        LLMResponse(
            tool_calls=(
                ToolCall(
                    id="call_1",
                    name="lookup_order",
                    arguments={"order_id": "A-123"},
                ),
            ),
            finish_reason="tool_calls",
        ),
        LLMResponse(
            content="Your order shipped yesterday.",
            finish_reason="stop",
        ),
    )

    def lookup_order(order_id: str):
        assert order_id == "A-123"
        return {"status": "shipped", "carrier": "UPS"}

    tools = {
        "lookup_order": Tool(
            name="lookup_order",
            schema={
                "type": "function",
                "function": {
                    "name": "lookup_order",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "order_id": {"type": "string"},
                        },
                        "required": ["order_id"],
                    },
                },
            },
            fn=lookup_order,
        )
    }

    answer, events = run_agent(
        fake,
        tools,
        [{"role": "user", "content": "Where is order A-123?"}],
    )

    assert answer == "Your order shipped yesterday."

    tool_events = [e for e in events if e["type"] == "tool_result"]
    assert len(tool_events) == 1
    assert tool_events[0]["name"] == "lookup_order"
    assert tool_events[0]["arguments"] == {"order_id": "A-123"}
    assert tool_events[0]["result"]["status"] == "shipped"

This test verifies:

• the model interface was called,
• the agent selected the right tool,
• the tool arguments were correct,
• the tool result was fed back,
• the agent terminated,
• the final response was handled.

No real LLM required.

Why this works: The event list becomes your observable trace. You can assert sequencing, detect loops, verify that dangerous tools were not called, and confirm that required tools were called before side effects.

A useful pattern is to assert negative behavior too:

def test_agent_does_not_call_refund_tool_without_verification():
    # scripted model tries refund immediately
    ...
    tool_names = [e["name"] for e in events if e["type"] == "tool_result"]
    assert "process_refund" not in tool_names

This is often more valuable than asserting the happy path.

---

6. Simulate the Model's Bad Days

Scenario: Your test suite only covers the case where the model returns perfect JSON and valid tool calls. Then production hits a response with truncated JSON, an unknown tool name, or a network timeout. Your agent crashes or loops.

Why it matters: A deterministic test harness is most valuable when it tests failure modes you cannot reliably provoke with a real model.

Your fake LLM can return exceptions, malformed output, invalid tool calls, or partial responses.

Timeouts and transient errors

class TransientLLMError(Exception):
    pass

fake = script(
    TransientLLMError("timeout"),
    LLMResponse(content="Recovered."),
)

Then test your retry wrapper:

def complete_with_retry(
    llm: LLMBackend,
    messages: list[dict[str, Any]],
    tools: list[dict[str, Any]] | None = None,
    attempts: int = 3,
) -> LLMResponse:
    last_error: Exception | None = None

    for attempt in range(attempts):
        try:
            return llm.complete(messages, tools)
        except TransientLLMError as exc:
            last_error = exc

    raise RuntimeError("LLM retries exhausted") from last_error

Test:

def test_retry_recovers_after_transient_error():
    fake = script(
        TransientLLMError("timeout"),
        LLMResponse(content="Recovered."),
    )

    response = complete_with_retry(fake, [])

    assert response.content == "Recovered."
    assert len(fake.calls) == 2

Unknown tool calls

Models sometimes invent tools. Your harness should verify that the system handles this safely.

def test_unknown_tool_is_rejected():
    fake = script(
        LLMResponse(
            tool_calls=(
                ToolCall(
                    id="call_1",
                    name="delete_all_data",
                    arguments={},
                ),
            ),
            finish_reason="tool_calls",
        ),
        LLMResponse(content="I could not perform that action."),
    )

    answer, events = run_agent(fake, {}, [{"role": "user", "content": "Do it."}])

    tool_events = [e for e in events if e["type"] == "tool_result"]
    assert tool_events[0]["result"]["error"] == "unknown_tool"
    assert "could not" in answer.lower()

Malformed structured output

If your app expects JSON, test what happens when the model gives you almost-JSON.

def parse_model_json(text: str) -> dict[str, Any]:
    try:
        return json.loads(text)
    except json.JSONDecodeError as exc:
        raise ValueError("Model returned invalid JSON") from exc

def test_invalid_json_raises_controlled_error():
    fake = script(
        LLMResponse(content='{"refund_amount": 25.0,'),
    )

    response = fake.complete([])

    try:
        parse_model_json(response.content or "")
        assert False, "Expected ValueError"
    except ValueError as exc:
        assert "invalid JSON" in str(exc)

If your application has a repair step — for example, asking the model to retry with valid JSON — script the bad response followed by the good response.

fake = script(
    LLMResponse(content='{"refund_amount": 25.0,'),
    LLMResponse(content='{"refund_amount": 25.0}'),
)

Why this works: You are no longer waiting for production to discover the model’s failure modes. You can encode them as regression tests.

🚨 Production warning: If your agent can retry, make sure your tests verify the retry limit. Infinite retry loops are one of the most common agent failures I’d worry about.

---

7. Replay Production Traffic Without Paying for It

Scenario: A real customer asks something weird: “Can you refund the shipping but keep the subscription active and also email my accountant?” Your test fixtures never imagined this. A real LLM handles it reasonably, but now you want a regression test for the surrounding logic.

Why it matters: Hand-written fixtures cover the cases you expect. Production covers the cases you did not. Recording and replaying real requests gives you realistic inputs without paying for every test run.

The basic idea is simple:

1. Record a sanitized request and response pair.
2. Store it as a fixture.
3. Replay it in tests through the same LLMBackend interface.

A minimal replay implementation:

import hashlib
import json
from pathlib import Path

def request_fingerprint(messages: list[dict[str, Any]]) -> str:
    canonical = canonical_json(redact(messages))
    return hashlib.sha256(canonical.encode()).hexdigest()

class MissingRecording(Exception):
    pass

class ReplayLLM:
    def __init__(self, path: Path):
        self.index: dict[str, dict[str, Any]] = {}

        for line in path.read_text().splitlines():
            entry = json.loads(line)
            self.index[entry["fingerprint"]] = entry["response"]

    def complete(
        self,
        messages: list[dict[str, Any]],
        tools: list[dict[str, Any]] | None = None,
    ) -> LLMResponse:
        fingerprint = request_fingerprint(messages)

        if fingerprint not in self.index:
            raise MissingRecording(
                f"No recorded LLM response for fingerprint {fingerprint}."
            )

        raw = self.index[fingerprint]

        return LLMResponse(
            content=raw.get("content"),
            tool_calls=tuple(
                ToolCall(
                    id=call["id"],
                    name=call["name"],
                    arguments=call["arguments"],
                )
                for call in raw.get("tool_calls", [])
            ),
            finish_reason=raw.get("finish_reason", "stop"),
        )

A recorded fixture line might look like:

{
  "fingerprint": "d2f1...",
  "response": {
    "content": null,
    "tool_calls": [
      {
        "id": "call_1",
        "name": "lookup_subscription",
        "arguments": {
          "customer_id": "cust_42"
        }
      }
    ],
    "finish_reason": "tool_calls"
  }
}

This is powerful, but it comes with caveats.

First, scrub everything sensitive before recording. Names, emails, account IDs, tokens, internal URLs, and PII should be replaced with stable placeholders. If you cannot scrub reliably, do not record.

Second, replay tests are only deterministic if the fingerprint is stable. If your prompts include timestamps, random IDs, or non-deterministic ordering, the fingerprint will change every run. Normalize those before hashing.

Third, recorded model responses are not “truth.” They are one possible behavior from one model version at one time. Use replay to test your system’s handling of that response, not to certify that the response is ideal.

Why this works: Replay gives you realistic edge cases while keeping CI offline, fast, and cheap.

💡 Practical note: Keep recordings versioned with your test suite. If a prompt changes intentionally, update the recordings deliberately instead of silently regenerating them.

---

8. Add Deterministic Evaluators That Do Not Need a Model

Scenario: Product says, “The assistant must never promise a refund before verifying the order.” You do not need a real LLM to check that. You need a rule.

Why it matters: Many AI quality checks are not semantic mysteries. They are policy constraints. Deterministic evaluators are cheaper, faster, and easier to debug than model-based judges.

Useful deterministic evaluators include:

• schema validators,
• regex and keyword checks,
• prohibited-phrase detectors,
• tool-trace validators,
• state-change validators,
• length and formatting checks,
• locale or tone constraints that can be expressed as rules.

Example: a refund policy evaluator.

import re
from typing import Any

def evaluate_refund_response(
    answer: str | None,
    events: list[dict[str, Any]],
) -> list[str]:
    errors: list[str] = []

    if answer is None:
        return ["No final answer was produced."]

    lowered = answer.lower()

    if re.search(r"guarantee|promise|definitely", lowered):
        errors.append("Response makes an unsupported guarantee.")

    if "refund" in lowered:
        tool_names = [
            event["name"]
            for event in events
            if event["type"] == "tool_result"
        ]

        if "verify_order" not in tool_names:
            errors.append("Refund mentioned without verifying the order.")

    if len(answer) > 1000:
        errors.append("Response is too long for the support UI.")

    return errors

Then use it in a test:

def test_refund_answer_follows_policy():
    fake = script(
        LLMResponse(
            tool_calls=(
                ToolCall(
                    id="call_1",
                    name="verify_order",
                    arguments={"order_id": "A-123"},
                ),
            ),
            finish_reason="tool_calls",
        ),
        LLMResponse(
            content="Your refund is approved and should arrive soon.",
            finish_reason="stop",
        ),
    )

    tools = {
        "verify_order": Tool(
            name="verify_order",
            schema={},
            fn=lambda order_id: {"eligible": True},
        )
    }

    answer, events = run_agent(
        fake,
        tools,
        [{"role": "user", "content": "Can I get a refund?"}],
    )

    errors = evaluate_refund_response(answer, events)
    assert errors == []

This approach also works well for structured output. If the model must return a JSON object, validate it with a schema or a Pydantic model.

With Pydantic v2, you might do:

from pydantic import BaseModel, Field, ValidationError

class RefundDecision(BaseModel):
    refund_amount: float = Field(gt=0)
    currency: str = Field(min_length=3, max_length=3)
    reason: str

def validate_refund_decision(payload: dict[str, Any]) -> RefundDecision:
    return RefundDecision.model_validate(payload)

Then test invalid payloads:

def test_invalid_refund_amount_is_rejected():
    try:
        validate_refund_decision({
            "refund_amount": -5,
            "currency": "USD",
            "reason": "Damaged item",
        })
        assert False, "Expected validation error"
    except ValidationError:
        pass

Why this works: Deterministic evaluators turn fuzzy product requirements into executable checks. They are not perfect, but they are debuggable. When one fails, you can usually see exactly why.

Model-based evaluation still has a place, especially for subjective quality. But if you put an LLM judge in the middle of every CI run, you have replaced one nondeterministic system with another. If you must use a model judge, run it offline, cache its judgments, and version them like fixtures.

---

9. Put Real LLMs in a Nightly Canary, Not CI

A deterministic harness does not mean you never call a real model again. It means you stop using real model calls as the primary gate for ordinary code changes.

A useful split looks like this:

Test layer	Uses real LLM?	Determinism	Speed	Cost	Best use
Unit tests with fake LLM	No	High	Very fast	None	Parsing, routing, retries, policies
Integration tests with scripted traces	No	High	Fast	None	Agent loops and tool orchestration
Replay tests	No	High if scrubbed and pinned	Fast	Storage only	Regression against production-like inputs
Local small model smoke tests	Optional	Medium	Medium	Compute only	Sanity checks for prompt formatting
Nightly real-model evaluation	Yes	Low	Slow	Real	Quality trends, model upgrades, prompt drift

The nightly canary is where you answer different questions:

• Did the new prompt reduce tool-call accuracy?
• Did a model version change alter response style?
• Are refusals increasing for valid requests?
• Are structured outputs still valid across a sample set?
• Did latency or token usage change materially?

Those questions are important, but they are not ideal for every pull request. They are slow, expensive, and noisy.

For the canary, use a fixed evaluation dataset. Keep it versioned. Include examples that previously failed. Record metrics over time instead of treating each run as a pass/fail oracle.

A practical canary structure:

evals/
  datasets/
    support_basic.jsonl
    support_edge_cases.jsonl
    refund_policy.jsonl
  metrics/
    tool_accuracy.py
    schema_validity.py
    policy_violations.py
  reports/

Each dataset entry can contain:

{
  "input": "Where is order A-123?",
  "expected_tools": ["lookup_order"],
  "must_not_call": ["process_refund"],
  "policy_checks": ["no_guarantees", "mentions_refund_only_if_verified"]
}

Then your real-model run scores the responses against deterministic checks. The model produces the behavior; your harness measures it.

This is the right place for real LLMs. They provide realism. Your deterministic harness provides engineering discipline.

---

What I Would Actually Ship

If I were building this for a production agent in 2026, I would not try to make one perfect test framework. I would assemble a few layers that each solve a specific problem.

1. A model port

All model access goes through one interface. No provider objects in business logic. No ad hoc SDK calls scattered across services.

2. A scripted fake for fast tests

Every agent loop, parser, retry policy, and tool-permission check gets unit tests with a scripted LLM. These tests run on every commit.

3. Trace assertions

I would not only assert the final answer. I would assert:

• which tools were called,
• in what order,
• with what arguments,
• which tools were not called,
• whether the loop terminated,
• whether state changes happened after required checks.

4. Golden prompt snapshots with redaction

Prompt changes should be visible in review. If a system prompt changes, I want a diff. But I would normalize dynamic values first so timestamps and IDs do not create noise.

5. Failure-mode tests by default

Every agent feature gets tests for:

• malformed JSON,
• unknown tool,
• invalid tool arguments,
• transient provider error,
• retry exhaustion,
• policy violation,
• loop limit exceeded.

These are the bugs that wake people up.

6. Replay for production surprises

When production reveals a weird input, I would scrub it, record it, and turn it into a regression test. This is how the harness gets smarter over time.

7. A separate real-model eval pipeline

Real LLM calls belong in a scheduled evaluation process, not in every unit test. That pipeline tracks trends, not just green checkmarks.

---

The final architectural point is this: a deterministic AI test harness is not a way to test whether the model is intelligent. You cannot unit-test intelligence, and you should not try.

It is a way to test whether your product behaves correctly when the model says the right thing, says the wrong thing, says nothing, says too much, calls a tool it should not call, or disappears behind a timeout.

That is the part you can control. That is the part you should test.