The review had four action items. Two of them said "update the system prompt to clarify that the assistant must not cancel orders that have already shipped." One said "add a warning to the tool description." The fourth was "monitor."
All four shipped within a day. Three weeks later the same failure happened again, from a different angle: this time the model didn't cancel a shipped order, it issued a partial refund on one, because the refund tool had no shipped-order precondition and the prompt's new sentence was about cancellation. The second review produced the same kind of action item, phrased slightly differently.
This is the normal trajectory of AI incident reviews. The review happens, everyone agrees the model was confused, someone writes a sentence into the prompt, and the system becomes marginally more brittle. The prompt grows, the instruction set develops internal contradictions, and the failure class survives because the actual defect was in the tool contract — a tool that let the model express an invalid action at all.
Why AI postmortems stall where normal ones don't
A conventional incident review has a gift: a stack trace with a line number. The failure is an event with a location, and the fix is usually local.
An AI incident gives you a plausible output. The system did something wrong and there is no line where wrongness happened. What you have instead is a trajectory: a prompt, a retrieval result, a sequence of tool calls, a model response, and a set of ambient conditions — model snapshot, temperature, context length, prompt version — that produced it.
Three properties make the review harder.
The failure is a distribution, not an event. One bad output might be a 0.5% behavior that existed for months, or a 40% behavior introduced this morning. Without a rate, you can't tell whether you're looking at a regression or a long-standing tail. The first job of the review is to convert an anecdote into a rate.
The "code" is prose plus schema. Your two deployable artifacts are a prompt (natural language, no type system, no compiler) and a set of tool definitions (structured, but interpreted by a model that ignores constraints it finds inconvenient). Neither has a diff that shows semantic change. A one-word prompt edit can invert behavior across the whole product.
The fix can make things worse in ways tests don't catch. Adding a rule to a prompt can break three unrelated behaviors. Tightening a tool schema can make the model fail to call the tool at all. Both changes need evaluation, and most teams don't have an evaluation that would catch the side effects.
🚨 Production warning: an incident review that closes with only prompt edits has a high probability of recurrence within a quarter. Prompt edits are the weakest durable fix available, and they're the one everyone reaches for first because they're fast to ship.
Capture the evidence before you need it
You cannot review what you didn't record. The review's quality is capped by the telemetry decisions you made months earlier.
The minimum evidence set for a single bad decision:
| Item | Why it's needed |
|---|---|
| Trace ID and full span tree | Reconstructs the sequence, not just the final answer |
| Prompt template ID and content hash | Identifies exactly which instructions were live |
| Model name and snapshot | Providers change behavior under a stable alias |
| Decoding parameters | Temperature and max tokens explain variance |
| Retrieved document IDs and versions | Distinguishes "bad reasoning" from "bad context" |
| Tool schema version | The contract the model was working against |
| Tool call arguments and results | Where the invalid action was expressed |
| Outcome and downstream effect | Severity, and whether the user saw it |
| Prompt version deployed at the time | Needed if you deploy prompts independently of code |
| Evaluation scores for that prompt version | Whether this was known-bad before launch |
Two of these are routinely missing and both are cheap to add. The first is the prompt content hash: if you store the template by name only, you can't tell whether the incident happened before or after Tuesday's edit. Hash the rendered template, store the hash on every trace, and keep the template versions addressable.
The second is the tool schema version. Teams version their code and their prompts, then let tool definitions drift because they live inside the prompt-building function. When the review asks "did the model have a reason parameter on cancel_order at the time?", nobody can answer.
Once you have the evidence, the first useful act is not analysis. It's a query: how many traces in the last 30 days exhibit the same failure signature? That number decides whether you're doing a regression review or a design review.
Classify the defect before you fix it
The single most valuable discipline in an AI incident review is refusing to accept "the model misunderstood" as a diagnosis. It's a description. The defect is somewhere upstream, and there are four places it usually lives.
| Symptom | Likely defect class | Durable fix |
|---|---|---|
| Model took an action it shouldn't have been able to take | Tool contract | Remove the capability or add a server-side precondition |
| Model chose the wrong action among valid ones | Prompt / policy ambiguity | Sharpen the decision rule, add a discriminating example |
| Model had the right instruction but wrong facts | Retrieval / data | Fix the source, the filter, or the freshness |
| Model behaved correctly; the product asked for the wrong thing | Product design | Change the workflow, not the model |
| Model failed only for a specific input shape | Input handling | Validate or normalize before the model sees it |
The table matters because the classes have wildly different durability. Removing a dangerous capability fixes the class forever. Adding a sentence to the prompt fixes the instance, and often creates an adjacent one.
A useful forcing question: if the model were replaced tomorrow with a different model, would the defect return? If yes, the defect is in your system, not in the model. If no, you're tuning around a specific model's quirk, which is legitimate but should be labeled as such — it becomes technical debt tied to a snapshot.
Prompt changes that survive the next six months
Prompt edits aren't useless. They're just load-bearing in a way that requires more rigor than a sentence appended to a wall of text.
Write the failing case first. Before editing the prompt, add the incident to your evaluation set as a concrete case with an expected behavior. This does two things: it proves the change works, and it prevents the next person from quietly removing your sentence during a cleanup.
Make the rule discriminating, not exhortative. "Be careful with refunds" is unenforceable. "If the order has a shipped_at timestamp, do not issue a refund; offer a return instead" is testable. Rules that name a field, a condition, and an alternative action are the ones that hold.
Check for collisions. Most mature prompts have a dozen rules, and new rules often contradict old ones at the edges. Before shipping, search the prompt for the concepts your new rule touches. If there's an existing instruction about refunds, you have three options: replace it, scope both rules explicitly, or delete the older one. Appending is the option that produces contradictory instructions and unpredictable behavior.
Put the rule where it will be read. Instruction-following degrades with distance and with volume. A rule buried at token 9,000 of a system prompt competes with everything after it. Keep the decision-critical rules near the top of the static block, where they're also inside the cached prefix.
Measure the blast radius. Run the full evaluation set, not just the new case. A prompt change that fixes one behavior and breaks two is a net regression, and it's the most common outcome of rushed prompt surgery.
🔍 Why this matters: a prompt is a shared mutable global that every feature reads. Treating an edit as a local fix is how you get a system whose behavior nobody can explain.
Tool contract changes: the fixes that actually hold
If the incident involved an action the system shouldn't have been able to take, the durable fix is at the tool boundary. Four levels, in increasing strength.
Level 1: constrain the arguments. Make invalid values unrepresentable. If a refund tool accepts an arbitrary amount, an enum plus a computed maximum derived server-side is stronger than a sentence in the prompt. If a date must be in the future, say so in the schema and validate it.
{
"name": "issue_refund",
"description": "Issue a refund for an order. Only valid when the order has not shipped.",
"parameters": {
"type": "object",
"additionalProperties": false,
"required": ["order_id", "reason_code"],
"properties": {
"order_id": { "type": "string", "pattern": "^ord_[A-Za-z0-9]{8,}$" },
"reason_code": {
"type": "string",
"enum": ["damaged", "not_received", "duplicate_charge", "goodwill"]
},
"amount_cents": { "type": "integer", "minimum": 1 }
}
}
}
Level 2: validate in the tool, not in the model. Every tool should re-check the preconditions it depends on, against the database, at execution time. issue_refund looks up the order, checks shipped_at IS NULL, checks the amount against the remaining balance, and returns a structured refusal if either fails. The model's belief about the order state is irrelevant.
Level 3: return errors the model can act on. A generic {"error": "invalid"} teaches nothing and produces retry loops. A structured error with a code, a human-readable reason, and a suggested alternative turns a failure into a recovery:
{
"ok": false,
"error": {
"code": "order_already_shipped",
"message": "Order ord_88213 shipped on 2026-09-14; refunds are not available.",
"alternatives": ["create_return", "escalate_to_human"]
}
}
Level 4: remove the capability. If a tool has caused two incidents, the question isn't how to constrain it, it's whether the agent should have it. Making issue_refund human-approval-only is a legitimate, boring, effective fix. So is deleting cancel_order and routing cancellation through a workflow with its own state machine and its own guards.
The strength ordering is the point. A prompt sentence is a suggestion to a probabilistic system. A schema constraint is a rejection at the boundary. A server-side precondition is an invariant. A removed capability is an impossibility. Incidents that caused customer harm deserve a fix at level 3 or 4, not level 1.
Version prompts and tool contracts like the code they are
None of this works if the artifacts are edited in place on a shared environment.
A prompt should live in version control as a file, ship through the same review process as code, and be identified at runtime by a content hash that appears in every trace. On Anthropic's API, cached prefixes are hashed across tools, system, and messages in that order, and changing tool definitions invalidates the entire cache. That has a practical consequence for review: a tool description edit is not a comment change, it's a cache-invalidating, behavior-changing deploy. Treat it as such and put it behind the same gate.
The gate itself should be short and automatic:
- Run the evaluation set. Fail on regression beyond a threshold you defined before the change.
- Diff the rendered prompt against the previous version. Reviewers should see the actual text change, not a template diff with placeholders.
- Deploy to a small canary cohort with the incident case included.
- Compare the canary's failure-signature rate to the baseline before widening.
That's four steps, and only the first requires machinery most teams don't already have. Even a fifty-case evaluation set is enough to catch the worst regressions, and building it is the highest-leverage thing an AI team can do after shipping.
Tool contract versioning, when other things depend on it
If only your own agent calls the tool, versioning is simple: change it, update the prompt, run the eval. The moment a second consumer exists — another agent, a partner integration, a customer's workflow — the contract becomes public.
Three rules that prevent most breakage:
Additive changes are safe; renames are not. Adding an optional parameter with a default is fine. Renaming reason to reason_code breaks anything that pattern-matches on the old name, including prompts you forgot about.
Never repurpose an existing value. If reason_code: "other" used to mean "unclassified," don't later use it for "goodwill." Consumers branch on values, and silent semantic changes are worse than removals because they don't fail loudly.
Deprecate with a window and a warning in the response. Keep the old field for a defined period, return a deprecated_fields array, and log which callers still use it. Then remove it on the announced date.
Turning one bad output into a rate
The most consequential sentence in an AI postmortem is a fraction. "Three of 1,842 refund attempts in the last 30 days contained a refund-after-ship action" tells you the severity, the exposure, and whether the fix worked. "A customer reported a wrong refund" tells you almost nothing.
Getting that fraction requires a failure signature: a mechanical definition of the bad thing, precise enough to query.
Good signatures are built from structured fields, not from reading text:
outcome = 'refund_issued' AND order.shipped_at IS NOT NULLtool_calls.name = 'send_email' AND tool_calls.result.status = 'rejected' AND retry_count > 2response.contains_pii = truefrom your output scannerescalation_reason = 'unknown' AND user_reopened_within_24h = true
Note that none of these require a model to judge them. That's the design goal: if a signature needs an LLM to evaluate, it's expensive, non-deterministic, and hard to defend in a review. Build the signature from data you already log, and reserve judge-based evaluation for the cases where no structured signal exists.
When you genuinely need a judge — tone regressions, subtle policy violations — two rules keep it honest. Calibrate it against a hundred human-labeled examples and report its precision and recall in the review, so nobody treats its output as ground truth. And keep the judge's prompt versioned alongside the system it's evaluating, because a judge whose criteria drift will show you a trend that isn't there.
The signature also gives you a detection story. If the query can run over the last thirty days in under a minute, add it to a weekly job and alert when the rate crosses a threshold. Most AI incidents are found by a customer; the goal is to move that discovery earlier by a few days, which is usually achievable with three or four signature queries total.
Three reviews, three different fixes
Abstractions get fuzzy here, so it's worth walking through how the classification changes the outcome.
Case one: the refund after shipment. The first review added a prompt sentence about cancellation. The recurrence added a refund path. The correct classification was tool contract: issue_refund had no precondition and no awareness of shipping state. The durable fix was two-part — a server-side check that returns order_already_shipped with alternatives, and removal of issue_refund from the agent's tool list in favor of a request_refund tool that creates an approval task. The prompt change remained, but as documentation of intent rather than as the control. Signature query: refund_issued AND shipped_at IS NOT NULL, which now returns zero because the path no longer exists.
Case two: the tone regression after a model upgrade. Support summaries became blunt after a provider snapshot change. No tool was involved and no rule was violated; the output was simply harsher. Classifying this as a prompt problem leads to "be warmer," which helps for a week. The actual defect was process: the model alias was unpinned, so the upgrade shipped without evaluation. The durable fixes were pinning the snapshot, adding a tone dimension to the evaluation set, and a canary gate for model changes. The prompt didn't change at all, which is the correct outcome and feels unsatisfying in the meeting.
Case three: the confidently wrong policy answer. The assistant cited a policy that had been superseded. The trace showed the retrieved document ID was three versions old. That's a data defect, not a reasoning one, and the durable fix was in the ingestion pipeline: delete-on-supersede, plus a freshness assertion in the retrieval test suite. A prompt instruction like "prefer the most recent policy" would have been worse than useless, because the model had no way to know which document was recent.
Three incidents, three different layers, and only one of them touched the prompt. That ratio is what a healthy review process looks like.
Rolling back a prompt is harder than rolling back code
Code rollback is a deploy. Prompt rollback is also a deploy, but the artifact has different blast radius and different semantics, and most systems make it awkward.
The awkwardness comes from coupling. If prompts are built in code, rolling back the prompt means rolling back the application, which also reverts unrelated changes. If prompts are stored in a database and edited by hand, you have no review and no diff, and rollback means finding the previous value in an audit table — if there is one.
The arrangement that makes rollback boring: prompts stored as versioned files, deployed independently of the application, addressable by hash at runtime, with the hash recorded on every trace. Rolling back is then a config change pointing at a previous hash, and you can verify the rollback by querying traces for the new hash and confirming the failure signature drops.
Two additional rules. Keep the last N prompt versions live and addressable, because you will want to A/B the previous version against the fix for a day. And never delete a prompt version that appears in traces you still retain, or your audit record becomes uninterpretable.
The review document itself
Keep it short. A page and a half is enough, and the structure should force the two questions that matter.
Incident: <one line>
Impact: <who, how many, how bad, detected how>
Rate: <x of y traces in window; signature definition>
Timeline: <detection → mitigation → resolution>
Contributing factors:
- prompt: <version, rule involved>
- tool contract: <schema version, missing precondition>
- data: <source, freshness, filter>
- product: <workflow assumption>
What made it hard to detect:
What made it hard to recover:
Actions:
1. <change> — class (prompt/tool/data/product) — owner — date — how we'll know it worked
Two fields are non-negotiable. Rate converts an anecdote into a measurement, and it's what tells you whether the fix worked. How we'll know it worked forces a metric, a test, or a signature query — something that would fail if the action were never done. "Update the prompt" has no such check. "Add case refund-after-ship-003 to the eval set; canary failure-signature rate below 0.1% for seven days" does.
Four anti-patterns worth naming in the meeting
The prompt as landfill. Every incident adds a sentence, nothing is ever removed, and the prompt grows to 8,000 tokens of accumulated scar tissue. The model's adherence to any individual rule drops as the rules multiply. Budget the prompt: if it grows past a threshold, the review should include a removal, not just an addition.
"The model was confused." A description, not a cause. Push until someone names the field, the instruction, or the missing precondition.
No counterfactual. If you can't say what the system would have done with the fix in place, you can't evaluate the fix. The counterfactual is usually a test case, and writing it is the actual work.
The unreviewed schema change. Tool definitions get edited in the same commit as a prompt tweak, under a message like "improve refund handling," and nobody notices the contract changed. Require tool schema changes to be their own diff with their own reviewer, because they have a different failure mode and a different blast radius.
What a good outcome looks like
The measure of an AI incident review isn't the document. It's the ratio of durable changes to prompt edits over time.
A team that's learning produces reviews where most actions are level 3 or 4 tool changes, eval cases, and data fixes, with prompt edits reserved for genuine policy ambiguity. A team that's stuck produces reviews where the prompt grows every month, the same class of incident recurs with different surface details, and the model is blamed each time.
The practical test at the end of any review: name the artifact that changed, name the test that would fail if the change were reverted, and name the person who owns it. If any of the three is missing, the action item is a wish. Prompts and tool contracts are the two places where AI systems are actually specified, and they deserve the same rigor as the code around them — diffs, review, tests, and rollback. Anything less and you're not fixing the system, you're annotating it.