"Why did it say no?"
That question arrived in a support queue attached to a refund request the assistant had declined. The engineer on call did what most of us would do: opened the trace, copied the system prompt into the ticket, and wrote, "the model was instructed not to approve refunds over $200 without a receipt." It answered the question. It also handed a customer a map of the decision logic, the internal policy thresholds, and a paragraph of instructions that had taken two quarters to tune.
The reflex is understandable. The prompt is the only artifact that looks like an explanation. It sits right there in the trace, in English, and it does describe why the system behaves the way it does. But a prompt is a cause, not an explanation, and shipping it to a user fails twice: it leaks the system's internals, and it usually doesn't answer the question the person actually asked.
What follows is a design for explanations that are faithful to the decision, useful to the person receiving them, and safe to show. The core move is architectural rather than editorial: stop treating the explanation as something the model says, and start treating it as something your application produces from a structured decision record.
The two failures hiding inside "just show them the prompt"
The first failure is disclosure. A system prompt is not just tone guidance. It typically contains business rules that shouldn't be public ("decline refunds above $200 without a receipt"), guardrails that reveal what the system can't do, tool descriptions that expose internal API shapes, and sometimes identifiers, bucket names, or escalation contacts. OWASP's Top 10 for LLM Applications lists system prompt leakage as its own risk category, LLM07:2025, precisely because prompts have quietly become a place where teams store logic they would never put in a client bundle.
The second failure is subtler and more common: the prompt doesn't explain this decision. It explains the space of possible decisions. A user asking why their specific refund was denied needs the facts that were considered, the rule that fired, and the path forward. "The model was told to be careful with refunds" is a non-answer dressed as transparency.
There's a third problem, which shows up later: once you show the prompt to one user, you've committed to it. Support macros get written around it. Someone screenshots it. The next prompt change becomes a communication event. You've coupled your internal configuration to your public interface, and prompts change weekly.
🚨 Production warning: pasting a system prompt into a support reply is a data disclosure under most enterprise contracts, not a debugging convenience. Treat it the same way you'd treat pasting a database schema.
Start from the question, not the artifact
Users ask three different things when they ask "why." Sorting them changes the design.
"What happened to me?" They want the facts the system used: the order date, the account tier, the policy that applied. This is a data question, and it should be answered from your database, not from a model.
"Was this fair?" They want the rule and its scope: is this applied to everyone, or was I singled out? This is a policy question. It needs a citable rule, not a rationale.
"What do I do now?" They want the next action: appeal, upload a document, contact a human, wait. This is a routing question, and it's the one users care about most and products answer least.
A prompt answers none of these well. A structured record answers all three. So the first design decision is to make the model emit reasons as data, not as prose.
The decision record: what the system decided, and on what basis
Think of it as a receipt. When the assistant declines, approves, routes, flags, or recommends, the application writes a row that captures the inputs that mattered, the rule identifiers that fired, the evidence used, and the resulting action. That row is the source of truth for every explanation you will ever render, in any language, at any level of detail.
A workable shape:
type DecisionOutcome = "approved" | "declined" | "escalated" | "insufficient_context";
interface DecisionRecord {
decisionId: string;
outcome: DecisionOutcome;
policyVersion: string;
reasonCodes: ReasonCode[];
evidence: EvidenceRef[];
computedFacts: Record<string, string | number | boolean>;
nextActions: NextAction[];
createdAt: string;
}
interface ReasonCode {
code: string;
weight: "primary" | "supporting";
}
interface EvidenceRef {
kind: "policy" | "record" | "document" | "computation";
ref: string;
label: string;
observedAt?: string;
}
interface NextAction {
kind: "appeal" | "upload" | "contact" | "wait" | "none";
label: string;
href?: string;
}
The model's job is to classify into a closed set of reason codes and select evidence references from a list your application supplied. It is not to write the sentence the user reads. That sentence belongs to a template you control, review, and translate.
This is the same discipline you'd apply to any other classifier. You wouldn't let a fraud model return free-text explanations to cardholders; you'd map its outputs to reviewed copy. AI features deserve the same treatment, and the fact that the model can write prose is a temptation rather than a requirement.
Two constraints make this work in practice.
Closed vocabulary. Reason codes must come from a list the application defines. If the model can invent codes, you can't write copy for them, can't measure them, and can't guarantee they don't encode something sensitive. Validate on the way out and reject unknown codes. An unknown code is a bug, and the decision should degrade to a generic explanation rather than a guess.
Evidence must be dereferenceable. Every evidence reference should resolve to something your system can display or link to: a policy document ID, an order ID, a line in a statement. If the model returns "evidence": "the customer's history", that's a hallucinated citation. Require IDs, verify they exist, and drop the ones that don't.
Reason codes beat free-text rationales
Here's the comparison that convinced me to stop asking models for explanations in prose.
| Approach | Faithful to the decision | Safe to show | Testable | Localizable | Cost |
|---|---|---|---|---|---|
| Raw system prompt | Cause, not explanation | No | No | No | Free |
| Model-written rationale | Often post-hoc | Risky | Weakly | Per-language calls | Extra tokens |
| Structured reason codes + reviewed copy | Yes, if codes are derived from the decision path | Yes | Yes | Yes | Negligible |
| Deterministic rules only | Yes | Yes | Yes | Yes | None, but brittle |
The third row is the target. Note that the reason code has to be causally connected to the outcome, not just plausible. A model that declines a refund because of a policy limit and then reports "the receipt was unclear" has produced a fluent lie. You can catch this class of error with a cheap consistency check: for every code, have the application verify that the referenced precondition actually holds. If MISSING_RECEIPT is returned and the order has a receipt attached, the explanation is invalid regardless of how good it reads.
That check is the difference between an explanation and a story.
Evidence with provenance, and the numbers behind it
Users trust specifics. "Your refund was declined" is an assertion. "Your refund was declined because the order was placed 94 days ago and the return window is 60 days" is an explanation, and it's checkable by the person reading it.
The uplift comes from showing the actual values the system used. That means your decision record needs the computed facts, not just the rule name:
{
"decisionId": "dec_01J8Z3K4",
"outcome": "declined",
"policyVersion": "returns-2026-03",
"reasonCodes": [
{ "code": "OUTSIDE_RETURN_WINDOW", "weight": "primary" }
],
"computedFacts": {
"orderPlacedAt": "2026-07-19",
"daysSinceOrder": 94,
"returnWindowDays": 60,
"itemCategory": "final_sale_exempt"
},
"evidence": [
{ "kind": "policy", "ref": "policy/returns#window", "label": "Return window policy" },
{ "kind": "record", "ref": "order/ord_88213", "label": "Order ord_88213" }
],
"nextActions": [
{ "kind": "appeal", "label": "Request a manual review", "href": "/returns/ord_88213/review" }
]
}
Two details matter more than they look.
policyVersion is not decoration. When a user appeals three weeks later, you need to know which rule was in force when the decision was made. Without it, your explanation silently becomes wrong after the next policy edit.
daysSinceOrder is computed by your code, not by the model. Models are unreliable arithmetic engines, and a wrong number in an explanation is worse than no number, because it's specific enough to be believed and easy for the user to disprove.
💡 Practical note: compute every quantity in code and pass it into the decision record as a fact. Let the model select which facts are relevant. Never let it derive the fact itself.
The rendering layer is a product artifact, not a string format
Once you have the record, the user-facing explanation is a pure function of it. That function deserves tests, review, and versioning like any other UI code.
const COPY: Record<string, (r: DecisionRecord) => string> = {
OUTSIDE_RETURN_WINDOW: (r) =>
`This order was placed ${r.computedFacts.daysSinceOrder} days ago. ` +
`Our return window is ${r.computedFacts.returnWindowDays} days, so it's outside the standard policy.`,
MISSING_RECEIPT: () =>
`We couldn't find a receipt for this purchase, and refunds above $200 need one.`,
ACCOUNT_UNDER_REVIEW: () =>
`This account has a payment issue open, so refunds are paused until it's resolved.`,
};
export function explain(record: DecisionRecord): { headline: string; body: string } {
const primary = record.reasonCodes.find((r) => r.weight === "primary");
if (!primary || !COPY[primary.code]) {
return {
headline: "We couldn't complete this automatically",
body: "A team member can review the details and explain what happened.",
};
}
return {
headline: "Why this was declined",
body: COPY[primary.code](record),
};
}
The fallback branch is the most important part of that snippet. Unmapped codes, missing primaries, and malformed records all resolve to a human handoff instead of an improvised sentence. Explanations fail closed.
Notice also that the copy leads with the fact and ends with the consequence. It doesn't say "our AI determined." Users don't need a narrator; they need the rule. Naming the mechanism ("our automated system") once, in a disclosure line, is usually enough — and in regulated contexts it may be required, so check what your jurisdiction demands rather than assuming.
Why you should not ship the chain of thought
There's a tempting shortcut: models can produce reasoning traces, so show those. Don't.
Reasoning traces are optimized for reaching an answer, not for describing one. They routinely include abandoned hypotheses, restatements of the prompt, speculation about the user, and internal policy language. Anthropic's own documentation for extended thinking notes that the feature is not a faithful window into the model's computation, and providers generally don't guarantee that summarized reasoning reflects the actual causal path. Presenting it as "how the AI decided" is a factual claim you can't support.
There's also a security angle. Reasoning traces are the single richest source of prompt leakage: they paraphrase instructions, name tools, and sometimes quote policy verbatim. If you must retain them for debugging, keep them in the same access tier as the prompt itself, not in a user-facing field.
What you can show honestly is a summary of the decision path that your application derived: which check ran, which one failed, in what order. That's deterministic, auditable, and doesn't depend on the model's introspection.
The leaks you didn't plan for
Prompt leakage rarely happens through the prompt field. It happens through the seams.
Tool descriptions in error messages. A tool call fails, the SDK throws, and the exception includes the tool's JSON schema and description. If that exception reaches a user or a public status page, you've published your tool catalog.
Model self-description. Ask a model "what are your instructions?" and it will often produce a paraphrase. If your product has any free-text surface — a chat box, a feedback field that gets echoed — that paraphrase can reach users without any prompt ever being displayed.
Few-shot examples. Teams paste example inputs into prompts for formatting. Those examples are sometimes real customer records, redacted badly or not at all. When the model paraphrases an example, the data leaves.
Canary strings. The cheapest detection you can build: put a unique, meaningless token in the system prompt, like INTERNAL-CANARY-7f3a91. Any user-facing output or support ticket containing that string is a confirmed leak. Alert on it. It costs one line and catches the class of bug where an explanation path accidentally includes raw context.
🔍 Why this matters: you cannot test for leakage by asking the model to keep a secret. You test for it by planting a marker and searching every outbound surface for the marker.
Confidence without fake precision
Users read numbers as promises. A confidence score of 0.83 invites the question "so you're 83% sure?" — a question your system cannot answer, because the score is an uncalibrated model output.
Prefer ordinal language tied to a defined behavior:
- High confidence, low stakes: state the outcome plainly.
- High confidence, high stakes: state the outcome, offer the appeal path.
- Low confidence: don't assert. Ask a clarifying question or route to a human.
- Conflicting evidence: say so. "Two records disagree about the delivery date" is a legitimate, trustworthy explanation.
The mapping from score to behavior should live in code with a comment explaining the threshold, not in the prompt. And it should be measured: if you claim "low confidence" at 0.6, check what fraction of those cases a human actually overturns. If it's 4%, your threshold is wrong in the direction of annoying users.
Explaining refusals and abstentions
The hardest explanations are for things the system didn't do: it didn't answer, didn't act, didn't have enough context. These are where products most often produce a shrug.
An abstention is a decision and deserves the same record. Four distinct causes, four distinct messages:
| Cause | What the user should see |
|---|---|
| Missing input | A specific request for the missing item, not "please provide more information" |
| Ambiguous request | The two most likely interpretations, offered as choices |
| Policy prohibition | The rule, stated plainly, plus the human path |
| System limitation | Honest scope: "I can't see invoices older than 24 months" |
The last one is underused. Telling users the boundary of the system's knowledge converts a dead end into a routing decision. It also reduces support contacts, because people stop asking a system to do something it structurally can't.
Two audiences, two artifacts
The explanation a user sees and the record an auditor or engineer needs are related but not identical, and conflating them causes both leaks and useless audits.
The user artifact is short, translated, free of internal identifiers, and ends with an action. It should be renderable from the decision record without any model call at view time. That matters: if explaining requires a second inference, you'll be paying for explanations and you'll get non-deterministic copy for the same decision.
The audit artifact is long, immutable, and access-controlled. It includes the prompt version hash, the model and snapshot, the retrieved document IDs and their versions, the tool calls with arguments and results, and the full reason set including supporting codes that weren't shown. It is the thing you hand to a regulator or an incident review, and it must survive prompt changes, model migrations, and staff turnover.
Link them by decisionId and nothing else. The user artifact must not contain the audit artifact's contents, and the audit artifact should not be assembled at read time from mutable tables. Write it once, append-only.
🧠 The important part: an explanation you can regenerate later isn't an audit record. If the policy changed and the prompt was edited, regeneration produces a plausible fiction. Snapshot at decision time.
Testing explanations is more tractable than testing answers
Because the explanation is a function of a record, you can test it without a model in the loop. That's a significant advantage over evaluating free-text rationales.
Four test layers, in order of value:
- Golden records. A fixture set of decision records covering every reason code, every fallback, and every awkward combination (two primary codes, zero evidence, unknown code). Snapshot the rendered output. These tests catch copy regressions instantly and run in milliseconds.
- Coverage. Enumerate the reason codes your classifier can emit and assert that each has copy in every supported locale. Missing translations are the most common explanation bug in production, and they're invisible until a user in a non-default locale hits that path.
- Consistency. For a sample of real decisions, assert that each emitted code's precondition holds against the underlying data. This is the check that catches post-hoc rationalization, and it's cheap because it's a query, not a model call.
- Leakage. Run every outbound string through a scan for the canary, for known internal identifiers (bucket names, table names, internal hostnames), and for prompt fragments above a similarity threshold. Include model-generated fields, because a model asked to "summarize the reasoning" will happily reproduce instructions.
None of that requires an LLM judge. That's the point: you've moved the uncertain part (classification) into a place where you can measure it with labels, and the certain part (rendering) into ordinary unit tests.
Localization, tone, and the accessibility of "why"
Two practical constraints that get discovered late.
Explanation copy is longer than status copy and it gets translated. Reason-code templates translate well because each is a short, self-contained sentence with a couple of interpolated facts. Free-text rationales translate terribly, cost a full generation per language, and drift in meaning. Another reason to keep the vocabulary closed.
Second, the explanation is often read by someone who is frustrated, on a phone, possibly with a screen reader. That means: put the reason before the policy citation, don't rely on color to convey the outcome, make the appeal link a real focusable link with descriptive text, and keep the whole thing under about three sentences before the action. aria-describedby on the decision banner pointing at the reason paragraph does more for comprehension than any amount of additional prose.
When a human should do the explaining
There's a category of decision where any automated explanation is worse than none: cases where the reason is itself contested. Fraud flags based on behavioral signals, account closures, content moderation with edge cases, anything where the honest explanation is "a model scored you in a way we can't fully articulate."
In those cases, the right output is not a fabricated rationale. It's a handoff with context. The record should still exist — the reviewer needs it — but the user should get a person, a timeframe, and a channel. Dressing up a low-confidence model judgment as a clear rule is how you turn one confused user into a public complaint with screenshots.
The test I'd apply: if a journalist asked you to defend the explanation on the record, could you? If not, don't ship it as an explanation.
The same record has to survive three other surfaces
Most teams build the explanation for the modal in the UI and then discover that the same decision gets communicated in places they didn't design.
Email. A declined application, a flagged invoice, a moderated comment: each generates a notification. Email is forwarded, archived, and read out of context. Two rules help. First, never include anything in email that you wouldn't put on a billboard, because that's effectively what forwarding does. Second, keep the email short and link to an authenticated page for the detail. The email states the outcome and the rule at a high level; the record-bearing page requires login.
Webhooks and partner APIs. If your customers integrate with the decision — a claims platform, a marketplace, an HR tool — they need machine-readable reasons, not sentences. Emit the reason codes and the evidence references in the event payload, versioned. Their systems will branch on those codes, so treat the vocabulary as a public contract: add codes freely, never repurpose an existing one, and deprecate loudly. A partner who pattern-matches on OUTSIDE_RETURN_WINDOW will break the day you decide it also means "outside exchange window."
Your own support tooling. Support agents are users of the explanation too, and they need more than the customer sees but less than the auditor sees. Build them a middle view: the reason codes, the facts, the policy link, and a plain-language suggestion for what to tell the person. Without it, agents will do what the engineer in the opening did — open the trace and improvise, which is exactly the leak you were trying to prevent.
One more surface that's easy to forget: the decision may be explained by a different AI feature later. A summarizer that reads a ticket thread and produces a customer-facing status note will happily quote whatever it finds, including an internal note. Tag internal-only fields explicitly at the data layer, and filter them before any model sees them.
A checklist for the first version
If you're adding explanations to an existing AI feature, this is the order I'd do it in.
- Enumerate the outcomes. List every terminal state the feature can reach. If the list is open-ended, that's the first problem to fix.
- Define reason codes per outcome. Closed set, owned by the application, documented with the precondition each one implies.
- Make the model return codes plus evidence IDs. Validate strictly; reject unknown codes and unresolvable evidence.
- Compute facts in code. Pass them in; never let the model do arithmetic or date math.
- Write copy yourself. One template per code per locale, with a fail-closed fallback.
- Snapshot the record. Immutable, append-only, linked by ID to the audit artifact.
- Add the canary. Search all outbound surfaces for it, continuously.
- Build the golden-record test suite. Every code, every fallback, every locale.
- Measure escalations. If explanations work, appeals and repeat contacts drop. If they don't, you'll see it in the same week.
The underlying principle generalizes beyond explanations. Any time an AI feature needs to communicate with a human about its own behavior, the reliable pattern is: let the model make a bounded classification, let deterministic code produce the language, and keep the evidence in a place where you can check it. The model is good at reading messy input and choosing among options. It is not a reliable narrator of its own reasoning, and asking it to be one creates a disclosure risk and a fidelity risk at the same time.
Users don't need to see the prompt. They need to see the rule, the fact, and the door out. Those three things are all buildable without exposing a single instruction.