Every dashboard is green. p99 latency is 1.4 seconds, the error rate is 0.08%, the queue is empty. And for three weeks the assistant has been giving customers slightly worse answers — shorter, more hedged, occasionally citing the wrong policy — which nobody noticed until a renewal conversation went badly.
Nothing in your observability stack was going to catch that. Prometheus knows how long the request took. It doesn't know whether the answer was any good. And "was the answer good" is not a metric you can bolt onto a request counter; it's a separate measurement problem with its own sampling design, its own cost, and its own privacy constraints.
That gap is what people mean by AI observability, and it's usually described badly. It isn't "observability, but for AI." It's a different set of questions, some of which share infrastructure with application observability and most of which don't.
What application observability actually answers
It's worth being precise about what the existing stack does well, because the temptation is to declare it insufficient and rebuild something worse.
Application observability answers: is the system doing what it was told, at the expected speed, without errors? Request rate, error rate, duration, saturation. Traces that show which service called which and where the 800ms went. Logs with correlation IDs. For an LLM-backed feature, all of that still applies and all of it is still necessary — a model call is an I/O operation with a duration and a failure rate, and it deserves a span.
What it cannot answer:
- Whether the input was the kind of input the system was designed for.
- Whether the output satisfied the user's intent.
- Whether quality changed after a deploy, a model update, or a shift in user behavior.
- Whether the cost per successful outcome moved.
- Which decision the model made, and on what evidence.
Those are all questions about semantics, and the existing stack has no vocabulary for them. A request that returns 200 with a fluent, incorrect answer is indistinguishable from a correct one at every layer below the model.
🧠 The important part: application observability measures whether the machine ran. AI observability measures whether the run was worth anything. The second only exists if you deliberately record it, because correctness leaves no trace in a status code.
Four things you have to add
The additions cluster into four categories, and each has a different implementation cost and a different failure mode if you skip it.
1. Input distribution
You need to know what's arriving, not just how much. A sudden shift in the kinds of questions users ask is the leading indicator for almost every quality incident — new product launch, a support macro that redirects traffic, a competitor's outage sending users your way.
Don't log raw inputs by default (privacy, cost, volume). Log a compact characterization:
def characterize(prompt_messages: list[dict], retrieved: list[dict]) -> dict:
user_text = prompt_messages[-1]["content"]
return {
"chars": len(user_text),
"lang": detect_lang(user_text), # cheap heuristic or small model
"intent_bucket": classify_intent(user_text), # fixed small taxonomy
"has_attachment": bool(re.search(r"\[file:", user_text)),
"retrieved_count": len(retrieved),
"retrieved_top_score": round(retrieved[0]["score"], 3) if retrieved else 0.0,
"empty_retrieval": not retrieved,
}
Six numbers, no content. That's enough to detect a distribution shift, and it's cheap enough to compute on every request. The empty_retrieval flag alone catches a surprising number of incidents — a reindex that failed, a filter that excluded a tenant, a query-rewriting step that started returning nothing.
2. Output quality, measured on a sample
You cannot score every response. You can score a stratified sample and interpret it correctly, which is a different skill than adding a metric.
Three sampling strata that cover most needs:
- Random. Unbiased estimate of overall quality. Small (0.5–2% of traffic) and boring, which is what makes it useful.
- Suspicious. Responses that trip a heuristic — very short output, refusal text, schema repaired, low retrieval score, retry count above zero. Oversample these heavily; they're where the failures are.
- Stratified by intent. Equal numbers per intent bucket, so a low-volume but important category doesn't disappear into the average.
Score with a rubric, and score with a model whose agreement with humans you've calibrated on a few dozen examples. The output is not "accuracy: 0.91." It's "quality by intent bucket, with an interval, week over week," which is what actually tells you something changed.
def quality_report(scores: list[dict], week: str) -> dict:
by_bucket: dict[str, list[float]] = {}
for s in scores:
by_bucket.setdefault(s["intent_bucket"], []).append(s["score"])
out = {"week": week, "buckets": {}}
for bucket, vals in by_bucket.items():
n = len(vals)
mean = sum(vals) / n
# Wilson interval is stable for small n; don't report a bare mean.
z = 1.96
p = mean
denom = 1 + z**2 / n
centre = (p + z**2 / (2 * n)) / denom
margin = z * ((p * (1 - p) / n + z**2 / (4 * n**2)) ** 0.5) / denom
out["buckets"][bucket] = {
"n": n,
"mean": round(mean, 3),
"ci": [round(centre - margin, 3), round(centre + margin, 3)],
}
return out
The interval is not decoration. With a 1% sample on 20,000 daily requests, you have 200 responses a day — enough to detect a large regression within a week and nowhere near enough to detect a small one in a day. Reporting a point estimate invites people to react to noise, which produces prompt changes that make things worse and get reverted a week later.
3. Cost per outcome
Token cost is easy and almost useless on its own. The number that governs decisions is cost per successful task, because it captures retries, extra calls, and the heavier model you fell back to.
def cost_per_success(spans: list[dict]) -> dict:
total = sum(
s["input_tokens"] * IN_RATE[s["model"]] / 1e6
+ s["output_tokens"] * OUT_RATE[s["model"]] / 1e6
for s in spans
)
ok = sum(1 for s in spans if s["outcome"] == "accepted")
return {
"requests": len(spans),
"total_usd": round(total, 2),
"accepted": ok,
"usd_per_accepted": round(total / max(ok, 1), 4),
"attempts_per_accepted": round(len(spans) / max(ok, 1), 2),
}
attempts_per_accepted is the underrated one. It rises before the bill does, because a model that fails structured output more often triggers retries that are individually cheap and collectively expensive.
4. The decision path
For anything agent-shaped, the thing you need at 2am is the sequence: which tools were called, in what order, with what arguments, and what came back. That's a trace, and it's where the OpenTelemetry GenAI semantic conventions are heading — attributes like gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.operation.name.
Those conventions have been evolving and are still marked as experimental in the spec, which means attribute names can change between versions. Treat them as a starting vocabulary, pin the version you implement, and map to your own stable schema internally so a spec rename doesn't break your dashboards.
from opentelemetry import trace
tracer = trace.get_tracer("ai.feature")
def traced_call(fn, *, model: str, prompt_version: str, **kwargs):
with tracer.start_as_current_span("gen_ai.chat") as span:
span.set_attribute("gen_ai.system", "anthropic")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("app.prompt_version", prompt_version)
try:
result = fn(**kwargs)
except Exception as exc:
span.set_attribute("app.outcome", "error")
span.record_exception(exc)
raise
span.set_attribute("gen_ai.usage.input_tokens", result["usage"]["input"])
span.set_attribute("gen_ai.usage.output_tokens", result["usage"]["output"])
span.set_attribute("app.outcome", result.get("outcome", "unknown"))
return result
The app. prefix on your own attributes is the part that matters. Vendor-specific and spec-defined attributes will churn; your own schema shouldn't.
Why quality dashboards fail
Almost every team that builds a quality dashboard builds one that nobody looks at, for three predictable reasons.
It aggregates. An average quality score across all intents hides the fact that one intent dropped 20 points while another improved. Aggregate dashboards are comfort devices.
It has no baseline. A score of 0.82 means nothing without the score from last month on the same sample design. Most dashboards start fresh after each instrumentation change, so the line resets and the history is lost.
It's disconnected from action. If the dashboard shows a drop, what does the on-call engineer do? If the answer is "investigate," it's not a dashboard, it's an alerting backlog item. Useful quality metrics connect to a specific lever: this drop is retrieval (empty_retrieval up), this one is prompt (schema failures up after prompt v14), this one is provider (latency up, output length down).
The fix is to organize by failure mode, not by model or intent alone. A small table of failure modes with a count each is more actionable than any score:
| Failure mode | Detection | Typical cause |
|---|---|---|
| Empty retrieval | retrieved_count == 0 |
Index stale, filter wrong, query rewrite failed |
| Schema repair | json_repaired == true |
Prompt/model mismatch, temperature too high |
| Refusal | Output matches refusal pattern | Prompt too restrictive, policy false positive |
| Truncation | output_tokens == max_tokens |
Context or limit too small |
| Protected token lost | Validation failure | Prompt drift, model change |
| Retry | attempt > 1 |
Transient provider issue or systematic format failure |
| User corrected | Next turn contains an edit or negation | Answer was wrong and the user fixed it |
Six of those seven are computable without any model call, from data you already have. That's the highest-value instrumentation in the whole system, and it's usually absent because it's less interesting than a quality score.
Privacy has to be designed in, not filtered later
Logging prompts and responses is the default in most observability tooling, and for AI features it's a liability. Customer text, internal documents, account details, sometimes credentials pasted into a chat box.
Three rules that work:
Redact at write time, in the process that produces the log. Not in a pipeline afterward, because the pipeline will have a bug and the raw data will have already landed somewhere.
import hashlib
import re
PATTERNS = [
(re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.]+\b"), "[email]"),
(re.compile(r"\b(?:\d[ -]?){13,19}\b"), "[card]"),
(re.compile(r"\b(?:sk|pk|ghp|xox[baprs])_[A-Za-z0-9]{16,}\b"), "[key]"),
(re.compile(r"\b\d{3}-\d{2}-\d{4}\b"), "[ssn]"),
]
def redact(text: str) -> tuple[str, str]:
digest = hashlib.sha256(text.encode()).hexdigest()[:16]
for pat, repl in PATTERNS:
text = pat.sub(repl, text)
return text, digest
Keep the hash. It lets you join two events with the same input (retry analysis, duplicate detection) without storing the input. That's often all you actually needed.
Store a pointer, not the payload, by default. Log prompt version, model, token counts, retrieval IDs, tool names, and outcome. Store the full text only for sampled or flagged requests, in a separate store with its own retention and access control.
Make retention explicit and short. Prompt text is the highest-risk data in the system and the least useful after a week. Thirty days is generous for debugging; ninety is a liability. Whatever you choose, the retention policy should be enforced by the storage layer, not by a cron job someone wrote once.
🚨 Production warning: trace backends frequently export to third-party SaaS by default. Adding an LLM span to an existing tracer can ship customer text to a vendor you never approved, without a code review that mentions it. Check the exporter configuration before the first deploy, not after the first audit.
What the support team needs is not a dashboard
There's a stakeholder whose needs are usually served last and matter most: the people who talk to customers about the feature.
They don't need traces. They need to answer, in under a minute, "why did the assistant say that." Which means a single searchable view keyed by the identifiers they actually have — conversation ID, customer ID, timestamp — that shows what the system retrieved, which model and prompt version ran, whether a retry happened, and whether the user later corrected it.
Building that view is mostly a projection of data you already have, and it changes the support dynamic significantly. Instead of "we'll look into it," the first responder can say "it pulled an outdated policy article from June, and the current one wasn't indexed." That's a specific, checkable statement, and it's the difference between a ticket that closes and one that escalates.
Two design notes for that view. Don't show raw prompts. They're confusing, they contain internal instructions, and they invite speculation about model psychology. Show the retrieval set, the tool calls, the version identifiers, and the outcome. Second, make the correction visible. If the user edited the answer or asked for something else, that's the most informative field on the page, because it's the closest thing to a ground-truth label that exists in production.
Alerting on the right things
The alerting split that works:
Page on: error rate, provider 5xx/429 spikes, p95 latency breach, empty retrieval rate above a floor, schema failure rate above a floor. These are unambiguous, actionable, and usually have a known runbook.
Ticket on: quality score drop beyond its interval for two consecutive days, cost per success up more than 30% week over week, new failure-mode bucket appearing, refusal rate tripling.
Dashboard only: input distribution shifts, output length drift, token efficiency changes. Interesting, not urgent, and paging on them trains people to ignore pages.
The distinction matters because quality signals are noisy by construction. A pager that fires on a quality score will fire on sampling variance, and after the third false alarm the alert gets muted, and then the real regression is invisible. Slow signals deserve slow channels.
What a good trace looks like
Concretely, one request should be reconstructible from its trace without reading application logs:
span: gen_ai.chat 1240ms
gen_ai.system anthropic
gen_ai.request.model claude-sonnet-4-20250514
app.prompt_version support_v14
app.prompt_hash 9f2c…
app.retrieval.count 6
app.retrieval.ids [doc_881, doc_902, doc_881_3, …]
app.retrieval.top_score 0.74
app.context_tokens 6120
gen_ai.usage.input_tokens 6480
gen_ai.usage.output_tokens 212
app.schema_repaired false
app.outcome accepted
│
└── span: tool.search_policies 180ms
app.tool.args_hash a17b…
app.tool.result_count 6
app.tool.truncated false
That's 15 fields and it answers the questions that matter during an incident: which prompt, which model, what evidence, did anything get repaired, how many tokens, and did the user accept it. None of it is customer text. All of it is cheap.
The prompt_hash is worth calling out. It's a hash of the rendered prompt template, not the content. When quality drops after a deploy, comparing prompt hashes across the boundary tells you immediately whether the prompt changed or something else did — a distinction that otherwise takes an hour of git archaeology.
Reconstructing a bad answer after the fact
The practical test of observability is whether you can explain a specific bad output a week later. Not a category of bad output — one specific conversation a customer escalated.
With the trace shape above, the reconstruction is: the request ID from the escalation, the span's prompt hash and prompt version, the retrieval IDs and their scores, the model and token counts, the outcome field. From there:
- Pull the retrieval IDs and re-fetch the chunks. If the indexed content has changed since, you'll see that immediately — and that's often the answer, because a reindex replaced the document the model actually used.
- Check the prompt version's diff against the previous release. If the hash matches the deployed version and the version didn't change, the prompt isn't the cause.
- Compare against the same prompt on the same inputs today. If today's output is different, something upstream moved: model snapshot, sampling parameters, context trimming.
- Look at the retrieval scores. A top score of 0.42 on a question the system answered confidently is a retrieval failure dressed as a generation failure, and it's the single most common root cause in RAG-adjacent features.
None of that requires storing customer text, which is the point of storing identifiers rather than payloads. You keep the ability to investigate while keeping the corpus out of your log store.
The step people skip is step 3. Re-running the exact prompt against the current model is the only way to separate "the prompt was bad" from "the model changed," and if you don't have the rendered prompt (or enough to re-render it deterministically), you can't do it. That's an argument for storing the rendered prompt for sampled requests, redacted, with short retention — the input side is usually less sensitive than the output side, and it's the part you need to reproduce behavior.
🔍 Why this matters: a system you can't re-run is a system you can only guess about. Replayability is the difference between a postmortem that ends in a fix and one that ends in "we added more logging."
Online and offline evaluation are different instruments
Both get called "eval," which causes endless confusion in planning meetings.
Offline eval runs a fixed set of cases against a candidate change, before deploy. It's fast, deterministic-ish, and it answers "did this change break anything we already knew about." It's a regression test. Its weakness is that its cases age: the distribution shifts, new failure modes appear, and a suite that was representative in March is nostalgic by October.
Online eval measures live traffic, after deploy. It answers "is the thing we shipped actually working for the people using it." Its weakness is that you can't replay it and you can't A/B everything, because the cost and the risk are real.
The loop that keeps both honest: online failures become offline cases. Every escalated conversation that reveals a new failure mode gets reduced to a test case within a week. That's the mechanism that prevents the offline suite from becoming a museum, and it's the single practice that separates teams whose quality improves from teams whose quality oscillates.
@dataclass
class RegressionCase:
source: str # 'incident:2025-09-14' or 'synthetic'
input: dict
must_contain: list[str] = field(default_factory=list)
must_not_contain: list[str] = field(default_factory=list)
rubric: str | None = None
The source field matters for maintenance. Cases derived from incidents are the most valuable and also the most likely to encode a one-off weirdness; tagging them lets you prune cases that no longer reflect the product without losing the ones that guard a real invariant.
Observability has a cost, and it's not just money
Every field you add to a span is paid for three times: storage, cognitive load, and the discipline of keeping it accurate.
Storage is the visible one. A trace with fifteen attributes on every LLM call in a system doing 50,000 calls a day is a real bill, and it's the bill that gets cut first — usually by dropping the attributes that mattered. Sampling whole traces (rather than fields) preserves analysis capability at lower cost: keep 100% of errors and suspicious traces, 5% of the rest.
Cognitive load is the bigger one. A dashboard with forty panels is a dashboard nobody reads. The useful default is one page with the failure-mode table, the quality trend with intervals, cost per success, and a link into traces. Everything else is a drill-down.
Accuracy is the one that decays silently. An outcome field is only as good as the code that sets it, and code that sets it tends to be added once and never revisited. Six months later half the spans say unknown and nobody notices, because unknown doesn't alert. A cheap guard: alert when the proportion of unknown outcomes exceeds a few percent, which is really an alert that your instrumentation drifted.
if unknown_ratio > 0.05:
alert("instrumentation:outcome_unknown", value=unknown_ratio)
That alert has caught more real problems in my experience than most quality alerts, because it fires when someone adds a new code path and forgets the annotation — the exact condition under which your quality numbers stop meaning anything.
Where the two disciplines meet
The useful framing isn't that AI observability replaces application observability. It's that they answer consecutive questions.
Application observability tells you the request completed. AI observability tells you whether it should have. A feature that's fast, error-free, and quietly wrong is fully observable by the first discipline and completely invisible to it.
Building the second one is less work than it sounds: six characterization fields per request, a stratified sample scored weekly, a failure-mode counter table, an outcome field on every span, and a trace that carries the identifiers a human would need. That's a few days of instrumentation and one recurring hour a week to read it. The alternative is finding out from a customer, which is the same information arriving later and costing considerably more.
The ordering matters if you're starting from nothing. Add the outcome field first — one enum on one span — because without it, nothing else you record can be attributed to success or failure. Then the failure-mode counters, which are pure logic and need no model. Then sampling and scoring, which need a rubric and a calibrated judge. Then the distribution metrics, which are the most fun to build and the least urgent. Teams that start with quality scoring tend to end up with an elegant measurement of something nobody has decided how to act on.
And whichever order you pick, keep the two disciplines in one place. When an incident review can show request latency and answer quality on the same timeline, the argument about whether the feature was "working" stops being about impressions and always becomes a concrete question about which line moved first — which is usually the only question in the whole review that produces a fix rather than a discussion about whether the feature was ever good.