How to test an AI feature when the output is never exactly the same

The test passes on your laptop, fails in CI, and passes again when you rerun it. The output is a perfectly good answer each time. Your snapshot test is telling you that the punctuation changed, not whether the feature still works.

Traditional software tests compare an expected result with an actual result. That works beautifully for a parser, a payment calculation, or a state transition. It works poorly when the output is generated by a model whose response can vary with sampling, backend updates, context ordering, and tool results. Replacing exact equality with “looks okay” is not a solution; it just hides the failure in a human impression.

Testing a nondeterministic AI feature means defining which properties must hold, measuring those properties over representative cases, and separating model behavior from the deterministic application around it. The goal is not to make the model deterministic. The goal is to make a release decision that remains defensible when two valid outputs use different words.

Start by separating output variation from behavior variation

Two answers can be textually different and behaviorally equivalent:

“Your order is scheduled to arrive Thursday.”

“It should reach you on Thursday.”

If both reflect the tracking record, the difference is harmless. But these outputs are not equivalent:

“Your order is scheduled to arrive Thursday.”

“Your order is scheduled to arrive tomorrow.”

The first pair differs in phrasing. The second pair differs in a claim about the world. A robust test suite should tolerate the first and fail the second.

Before writing tests, classify what can vary and what cannot:

Output property	Usually allow variation?	Example assertion
Exact wording	Yes	No exact string comparison
Required facts	No	The returned date matches the tracking fixture
Prohibited claims	No	Do not promise a delivery time absent from the source
Output shape	No, if consumed by code	Must satisfy the JSON schema
Tool choice	Often constrained	Must call lookup before answering an order-status request
Tone	Bounded variation	Must not insult, threaten, or use forbidden claims
Cost and latency	Bounded variation	p95 latency and per-task cost remain within budget

This distinction sounds obvious. Many flaky AI tests come from failing to make it. A snapshot asserts wording when the product contract is about facts; a vague human review tolerates a wrong fact because the wording is polished.

OpenAI’s evaluation guidance describes the problem directly: generative AI is variable, so traditional tests alone are insufficient. It recommends task-specific evaluations, production-shaped datasets, continuous evaluation, and human calibration rather than generic metrics or “vibe-based” checks. (OpenAI, evaluation best practices)

Write the contract before choosing the grader

For each feature, write the contract in terms of outcomes. A summarizer might need to preserve named entities, dates, decisions, and unresolved questions while avoiding unsupported conclusions. A classifier must return a label from a known set. A tool-using assistant must call the right tool with authorized arguments before it claims that an action occurred.

A compact contract for an order assistant could be:

Given a valid order ID and an order lookup result:
- the answer must state the status returned by the service;
- it may include the delivery date only when one is present;
- it must not claim the order was changed;
- it must not call refund or cancellation tools;
- if the order ID is missing, it must ask for it rather than invent one.

This contract exposes the different test techniques you need. The schema and tool restrictions are deterministic. Whether the answer accurately paraphrases the status may need a semantic grader. Whether the response is clear to a customer may require human review.

Do not ask one judge to grade everything. A single “quality score” that blends correctness, tone, safety, and completeness is difficult to debug. Keep separate dimensions so a regression tells you what broke.

1. Test deterministic boundaries with ordinary assertions

The model is only one part of the feature. Input validation, authorization, state changes, retries, output parsing, and side effects are ordinary software and should be tested with ordinary tests.

Use unit tests for:

• request validation and maximum input size;
• tenant and role checks before tool execution;
• schema parsing and rejection of unknown fields;
• idempotency and duplicate request handling;
• timeout, cancellation, and retry behavior;
• output rendering and escaping;
• fail-closed behavior when a model call or grader is unavailable.

Mock the model at the boundary. Give the application valid, malformed, incomplete, and adversarial model outputs, then assert exactly what the application does. If a JSON parser accepts an unknown account_id and executes a write anyway, that is a deterministic bug; it should not be hidden inside an AI eval.

Conversely, do not mock away the model in a system eval and then claim the end-to-end feature passed. You need both levels: fast deterministic tests around the application and a separate evaluation that actually exercises the model and its tools.

2. Use exact checks wherever the contract is exact

Exact matching is not obsolete. It is just appropriate only for exact properties.

Examples:

• the response parses as JSON;
• intent is one of billing, technical, account, or unclear;
• a required tool was called exactly once;
• a forbidden tool was never called;
• a generated SQL statement contains no second statement;
• a date equals the fixture’s date;
• a monetary value is within a specified decimal tolerance;
• the output contains no raw access token.

Use schema-based output modes when the provider and model support them, and still validate at the application boundary. A provider-level structured-output guarantee can reduce malformed output; it does not replace authorization or business-rule validation.

For code that consumes a model-generated enum, the test should assert behavior for both valid and invalid labels:

ALLOWED = {"billing", "technical", "account", "unclear"}

def route_prediction(payload: dict) -> str:
    label = payload.get("intent")
    if label not in ALLOWED:
        return "human_review"
    return label

def test_unknown_label_falls_back():
    assert route_prediction({"intent": "refund_everything"}) == "human_review"

def test_missing_label_falls_back():
    assert route_prediction({}) == "human_review"

These tests are stable because the assertion is about application behavior, not model wording.

3. Build a dataset around decisions, not prompts

A useful eval set is a collection of cases with enough context to determine whether the feature behaved correctly. It is not merely a folder of prompts and expected strings.

For each case, store:

• a stable case ID;
• the user input and relevant context;
• expected behavior or decision;
• required facts and prohibited claims;
• expected tool behavior, when applicable;
• risk category and feature slice;
• provenance and a short note explaining why the case exists.

Cover the distribution you expect and the failures you cannot afford. A customer-support dataset needs normal order questions, missing order IDs, mixed intents, stale account information, policy exceptions, non-English inputs, and prompt-injection attempts in user-provided content. If the feature has a human escalation path, include cases that must escalate and cases that must not. A dataset with only escalation positives can reward a system that escalates everything.

Keep three sets distinct:

• Regression cases: known failures you fixed; run on every relevant change.
• Release set: representative cases used for launch decisions.
• Discovery set: new, often ambiguous cases sampled from production for investigation.

Promote a confirmed production failure into regression coverage. Do not blindly append every new example to the release set; that set needs deliberate review so its meaning remains stable.

For a new feature, a small expert-reviewed seed set is enough to begin iteration. It is not enough to make a strong statistical claim about a rare slice. Scale the set based on the decision you need to make, and report how many examples support each result.

4. Grade semantics with criteria, not string similarity

String similarity metrics can be useful when wording similarity is the actual goal, such as constrained translation or extractive summarization. They are weak proxies for correctness in many product tasks. A fluent answer can be wrong with a high overlap score; a correct paraphrase can have low overlap.

For semantic properties, use one or more of:

• a reference-grounded check: does each claim follow from supplied source data?
• a claim checklist: are required facts present and forbidden claims absent?
• pairwise evaluation: which of two answers better satisfies a narrow rubric?
• a task-specific classifier: is the answer supported, unsupported, or contradictory?
• expert review for high-risk decisions and ambiguous cases.

LLM-as-judge can scale review, but it is another probabilistic system. Calibrate it against human-labeled examples, measure false passes and false failures separately, and audit disagreements. If the judge is too lenient on confident wording or prefers longer answers, the eval can improve while user outcomes get worse.

Keep the rubric narrow. “Is this answer factually supported by the retrieved order record?” is easier to calibrate than “Is this a good answer?” Require the judge to cite the relevant source text or fact IDs so a human can inspect the basis of a score. A rationale is not proof, but it makes disagreement diagnosable.

Calibration should include both acceptable and unacceptable answers, including near misses. A judge that sees only obviously wrong answers can appear accurate while missing the subtle, confident mistakes that matter in production. Re-run calibration when the rubric, judge model, or output format changes; those changes can alter agreement even if the feature itself is untouched.

As of October 2026, OpenAI’s documentation says its Evals platform is being deprecated: existing evals become read-only on October 31, with shutdown scheduled for November 30, 2026. That is a good reason not to make your test corpus and release gates inseparable from one vendor’s evaluation product. Keep cases, rubrics, and result records in portable formats even if you use a hosted eval interface. (OpenAI deprecations)

5. Repeat runs to measure variance, not to hide it

A single run answers, “What happened this time?” Repeated runs help answer, “How often does this behavior happen?”

For a stochastic feature, run high-risk and borderline cases multiple times using the same pinned model and settings. Record each run independently. If an answer passes 9 of 10 times, the feature is not simply “90% reliable”; the sample is small, and the failure may be correlated with a specific slice. Still, the variance is useful evidence that a one-run regression test would miss.

Do not average away catastrophic failures. If one of 20 runs issues an unauthorized tool call, the mean success score is not the right release signal. Report failure counts and severity, and gate high-impact actions separately. For low-risk wording variation, a pass-rate metric may be adequate.

Use fixed seeds only when the API supports them and when they help isolate changes. A seed is not a contract that guarantees identical output across backend updates. Record model identifiers, sampling settings, prompt versions, tool schemas, and provider metadata where available. Otherwise, a score change after a release cannot be attributed to the prompt, the model, or the serving stack.

6. Add property and metamorphic tests

Some outputs have no single gold answer, but relationships between inputs and outputs should still hold. Metamorphic tests check those relationships.

Examples:

• Adding an irrelevant sentence to a support ticket should not change the account ID selected by the system.
• Reordering two independent documents should not change the set of facts in a summary.
• Changing the user’s preferred format from prose to bullets may change presentation, but not the extracted invoice total.
• Removing a source citation should make the model less certain, not more certain.
• A tool result that says status: cancelled must never produce a response claiming status: shipped.

Property tests are especially useful for robustness because they let you generate many variations from a known case. They do not replace a golden set: a property can be wrong or too weak. Have a domain expert review the property and the transformations before treating it as a gate.

For agentic systems, test invariants over the whole trace. A final answer may look correct even though the agent accessed a resource outside the user’s scope or called a write tool unnecessarily. Assert that every tool call is authorized, every write has an idempotency key, and a handoff does not loop indefinitely.

7. Evaluate each architecture layer and the end-to-end task

A workflow with several model calls can fail in different places: classification, retrieval, extraction, tool selection, argument construction, or final response. If you evaluate only the final prose, you may not know which component regressed.

Create component-level checks for the parts that can be isolated, then evaluate the complete task using realistic tool fixtures. OpenAI’s current evaluation guide recommends mapping the sources of nondeterminism to the parts of the architecture that should be evaluated; tool choice and handoffs add failure modes beyond ordinary text generation. (OpenAI, evaluation best practices)

A good report includes:

Layer	Example metric	Example failure
Input handling	Valid-input and rejection rates	Oversized request accepted
Retrieval	Recall and source correctness	Stale policy returned
Model output	Required-fact pass rate	Wrong delivery date
Tool selection	Correct tool rate	Refund tool called for status lookup
Tool arguments	Argument validity	Wrong tenant ID
Workflow	Completion and handoff rate	Circular handoffs
Product outcome	Accepted resolution rate	User reopens ticket

The overall success metric should reflect the product’s actual unit of value. For a support feature, that might be a resolved issue without a correction or repeat contact, not “model output passed a rubric.”

8. Use online evaluation without turning users into test subjects

Offline evals cannot represent every real input. Production data reveals missing cases, but logging everything raw can create privacy and data-retention risks.

Set up a controlled feedback loop:

1. Log the minimum information needed to diagnose behavior: prompt/model version, result status, tool events, token usage, latency, and a correlation ID.
2. Sample cases based on risk and drift signals, not only random traffic.
3. Redact or minimize sensitive content before traces reach broad-access systems.
4. Have qualified reviewers label failures and ambiguous cases.
5. Add confirmed failure patterns to regression coverage.
6. Track user outcomes such as edits, cancellations, escalations, and repeat contacts.

Do not treat thumbs-up as a direct correctness label. Users may approve a wrong answer because they did not notice the error; they may reject a correct answer because it is inconvenient. Combine explicit feedback with downstream behavior and expert review.

Sampling needs an owner and a cadence. A daily random sample can expose broad regressions; a risk-triggered sample can inspect every failed payment or repeated escalation. If review only happens after someone complains, your dataset will overrepresent visible failures and miss silent errors that users never recognized. Record why a case was sampled so later analysis can distinguish a representative estimate from a targeted investigation. A stable sampling plan makes week-to-week changes interpretable instead of anecdotal.

Keep user consent, retention, and access controls aligned with the data collected for evaluation; the need to debug an output does not justify permanent storage of every private conversation.

For a major model or prompt change, shadow or canary the new version on a small slice. A shadow request should not execute side effects. A canary should have a clear rollback condition and an owner watching the metrics. Compare quality, latency, cost, tool behavior, and harmful failure rates—not just one aggregate score.

A minimal portable harness

The test harness should keep model invocation separate from grading. The following Python sketch assumes system.run(case) returns a structured result and grade(case, result) applies deterministic and semantic checks. Provider integration and the semantic grader are deliberately injected so the dataset can outlive a particular SDK.

from dataclasses import dataclass
from typing import Protocol

@dataclass
class Case:
    case_id: str
    input: dict
    expected_status: str
    required_facts: list[str]
    forbidden_tools: list[str]
    risk: str

@dataclass
class RunResult:
    answer: str
    status: str
    tool_calls: list[dict]
    latency_ms: int
    input_tokens: int
    output_tokens: int

class SystemUnderTest(Protocol):
    def run(self, case: Case) -> RunResult: ...

def grade_deterministic(case: Case, result: RunResult) -> list[str]:
    failures = []
    if result.status != case.expected_status:
        failures.append("wrong_status")
    called = {call["name"] for call in result.tool_calls}
    forbidden = called.intersection(case.forbidden_tools)
    if forbidden:
        failures.append(f"forbidden_tool:{','.join(sorted(forbidden))}")
    return failures

def run_case(system: SystemUnderTest, case: Case, semantic_grade) -> dict:
    result = system.run(case)
    failures = grade_deterministic(case, result)
    semantic = semantic_grade(case, result)
    return {
        "case_id": case.case_id,
        "risk": case.risk,
        "passed": not failures and semantic["passed"],
        "deterministic_failures": failures,
        "semantic_reason": semantic["reason"],
        "latency_ms": result.latency_ms,
        "input_tokens": result.input_tokens,
        "output_tokens": result.output_tokens,
    }

In a real harness, persist every result with the dataset hash, prompt version, model ID, grader version, code commit, timestamp, and per-attempt usage. Do not store only the aggregate. The individual failed cases are the information that tells you what to fix.

Common test mistakes

Mistake	Why it misleads	Better approach
Snapshot the entire response	Flags harmless wording changes	Assert facts, shape, and prohibited behavior
Use one judge score	Hides which contract failed	Keep correctness, safety, style, and tools separate
Test only happy paths	Misses ambiguity and exceptions	Include negatives, edge cases, and adversarial inputs
Run every case once	Misses variance	Repeat important cases and report run counts
Trust synthetic examples alone	Evaluates the model’s imagined user	Mix expert, historical, and production-shaped data
Compare one aggregate score	Hides slice regressions	Report by intent, risk, language, and tenant where applicable
Ignore cost and latency	Can “improve” quality by making the feature unusable	Gate quality alongside service budgets
Keep no run metadata	Makes regressions irreproducible	Version the entire evaluation stack

Read the score with its sample size

An eval score is an estimate from a sample, not a permanent property of the feature. If 90 of 100 examples pass, “90%” looks precise but its uncertainty is substantial: a rough 95% Wilson interval is about 82% to 95%. The interval narrows as the dataset grows, but doubling the number of examples does not halve the uncertainty. This is why a change from 90% to 92% on a small set is not automatically an improvement.

Slice counts matter even more. A 94% aggregate score can coexist with a 60% pass rate on non-English inputs if that slice is small. Always show numerator and denominator: 19/20, not only 95%. For safety-critical cases, report the actual failures and their severity rather than smoothing them into a rate.

When comparing two prompts or models, use the same cases and compare paired outcomes. If both versions pass most easy cases, the useful signal is where their decisions differ. Review those disagreements, especially when one version changes a tool call or a factual claim. Randomized order for human pairwise review can reduce the tendency to prefer whichever answer appears first.

Repeated model runs on the same case answer a different question from adding more cases. More cases improve coverage of the input distribution; repeated runs estimate output variance on a fixed input. For an unstable high-risk behavior, do both. Do not report ten repeated runs of one example as ten independent user cases.

A release gate that respects uncertainty

There is no universal score that makes an AI feature safe to ship. A sensible gate states the evidence and the consequence:

• exact schema and authorization tests pass;
• no forbidden tool calls occur in high-risk cases;
• core task quality meets a pre-agreed threshold on a versioned release set;
• confidence intervals or repeated-run results are reported where sampling matters;
• cost and latency fit product budgets;
• safety and fairness slices are reviewed independently of the aggregate;
• the canary has rollback conditions and a named owner;
• production failures have a path into regression coverage.

For an irreversible action, one observed unauthorized write may block launch even if 999 other cases pass. For a low-risk style suggestion, a small amount of output variation may be acceptable. The gate should follow consequence, not the convenience of a single score.

The practical standard is not “the model returns the same answer every time.” It is “the feature preserves the properties users depend on, handles uncertainty safely, and gives the team enough evidence to detect a regression.” Exact snapshots still belong around deterministic code. The model’s behavior needs evaluation. The product outcome needs measurement. Put those layers together, and nondeterminism becomes something you can manage rather than an excuse for tests that either flake or say nothing.