Building an AI feature your support team can actually debug

A support agent opens a ticket that says: "The assistant told my customer their refund was approved. It wasn't." The agent has the customer's email, a timestamp, and a screenshot of a chat bubble. What they don't have is any way to find out which request produced that bubble, what documents the model read, which tool it called, or whether the refund tool returned an error that the model politely ignored.

So the ticket goes to the engineering channel, where someone asks for the user ID, then the conversation ID, then the approximate time, then logs into the model provider's dashboard, then discovers the request was served by a fallback model nobody remembered was configured. Forty minutes later, the answer is "the tool timed out and the model hallucinated a confirmation." The customer has been waiting the whole time.

That is not a model problem. It is a product-surface problem. The feature was built with a chat UI, an API key, and a prompt, and nobody built the part where a non-engineer can reconstruct what happened. Support teams are the first responders for AI features, and most AI features give them a screenshot and a shrug.

This article is about designing the debugging surface as part of the feature, not as an internal tool bolted on after the first bad ticket. The goal is narrow and testable: a support agent with no ML background should be able to answer five questions about a specific request in under ten minutes, without opening a terminal.

The five questions support needs to answer

Every AI support investigation reduces to a small set of questions. If your system can answer them, most tickets resolve at tier one. If it can't, every ticket becomes an engineering interrupt.

1. What did the user actually see? Not what the model generated — what was rendered, after streaming, after client-side truncation, after any post-processing.
2. What did the model receive? The assembled prompt, the retrieved context, the conversation history that was included, and the tool definitions that were in scope.
3. What did the system do on the user's behalf? Every tool call, its arguments, its result, and whether it committed a side effect.
4. Which version of everything was involved? Prompt version, model and model snapshot, retrieval index version, feature flags, and application release.
5. Was this a bug, a policy decision, or a capability limit? These route to different teams and have different fixes.

Notice that none of these questions require understanding transformer internals. They require bookkeeping. That's the encouraging part: debuggability is mostly a data modeling and access-control problem, not a research problem.

The discouraging part is that most of the bookkeeping is skipped because the demo worked without it. In a demo, you are the person looking at the output, and you remember what you typed. In production, the person looking at the output is a support agent in a different time zone who has never seen your prompt.

Debuggability starts with an event log, not a log file

Application logs are optimized for humans reading a stream of lines during an incident. Support debugging needs something different: a queryable record keyed by the thing the user can provide, which is almost always an email, an order number, a conversation, or a timestamp.

The minimum viable structure is two tables: one row per AI run, and one row per event within that run.

create table ai_runs (
  id                uuid primary key,
  trace_id          text not null,
  conversation_id   uuid not null,
  tenant_id         uuid not null,
  user_id           uuid,
  surface           text not null,
  status            text not null,
  prompt_key        text not null,
  prompt_version    integer not null,
  model             text not null,
  model_snapshot    text,
  index_version     text,
  feature_flags     jsonb not null default '{}'::jsonb,
  app_release       text not null,
  input_tokens      integer,
  output_tokens     integer,
  cost_usd          numeric(10, 6),
  started_at        timestamptz not null,
  finished_at       timestamptz,
  error_code        text
);

create table ai_run_events (
  id           bigserial primary key,
  run_id       uuid not null references ai_runs(id) on delete cascade,
  seq          integer not null,
  kind         text not null,
  payload      jsonb not null,
  occurred_at  timestamptz not null,
  unique (run_id, seq)
);

create index on ai_runs (conversation_id, started_at desc);
create index on ai_runs (tenant_id, started_at desc);
create index on ai_run_events (run_id, seq);

The kind column is where the design work happens. A useful starting vocabulary:

• request_received — the user-visible input, the surface, the client version.
• context_retrieved — document IDs, scores, index version, and the filter that was applied.
• prompt_assembled — prompt key, version, a hash of the final string, and token counts per section.
• model_call — model, snapshot, parameters, latency, finish reason, token usage.
• tool_call — tool name, arguments, result summary, latency, error code.
• policy_decision — any guardrail or authorization outcome, including passes.
• response_rendered — the final text after post-processing, plus any truncation.
• user_feedback — thumbs, edits, retries, escalations, and the reason if offered.

The ordering matters. Support needs to see the timeline, because the interesting failures are usually ordering failures: the tool ran before the confirmation, the retrieval happened before the index refreshed, the fallback model answered after the primary timed out.

Two schema decisions pay for themselves immediately. First, store trace_id on the run and propagate the same value through your queue, worker, and any downstream service. If you already emit OpenTelemetry spans, the GenAI semantic conventions give you a shared vocabulary for model, tool, and token attributes, which means your traces and your support view can be the same data rather than two systems that disagree. Second, never mutate these rows. Corrections go in as new events with a kind like annotation, so a support investigation can't be invalidated by a later job rewriting history.

⚠️ Gotcha: if you put the whole prompt in a single jsonb blob on the run row, you will eventually have a 400 KB row that breaks your ORM, your replication, and your retention job at the same time. Prompts belong in object storage or a dedicated table keyed by hash, with the run holding a reference.

The support view is a product screen, not a SQL query

The single highest-leverage thing you can build is one internal page that takes a conversation ID or an email and renders a readable timeline. Not a Grafana dashboard. Not a link to the provider's logs. One page, with the run's inputs, outputs, tool calls, versions, and a plain-language summary of what happened.

Here's the shape that works: a header with the user, tenant, surface, timestamp, and status; then a vertical timeline of events; then a "what the model was told" panel with collapsible sections for system prompt, retrieved context, and history; then a "what the system did" panel listing tool calls with arguments and results side by side; then a footer with versions and a copyable bundle for escalation.

Two details make this page usable by non-engineers.

First, label the failure, don't just show the data. "Retrieval returned zero documents" is a label. A list of twelve empty context_retrieved events is a puzzle. Compute a small set of derived explanations at read time and display them prominently:

Signal in the run	Support-facing label	Likely owner
context_retrieved with 0 results	No knowledge matched this question	Content / retrieval
Tool result with error_code then a normal answer	Tool failed; answer may be unsupported	Engineering
finish_reason = length	Answer cut off by token limit	Engineering
Guardrail block	Blocked by policy	Trust & safety
Same tool called 3+ times with identical args	Agent loop	Engineering
Model snapshot differs from default	Fallback model answered	Engineering
User edited or retried within 30s	User rejected the first answer	Product

Second, show the user's exact words next to the model's exact words. Support agents are very good at spotting the mismatch between "I was charged twice" and an answer about a single charge. They don't need to understand tokenization to notice the assistant answered a different question.

If you can only build one thing this quarter, build this page. It converts a class of tickets from "needs an engineer" to "needs a support agent with a link."

Correlation IDs that survive streaming, queues, and retries

The most common reason a support view is useless is that the ID the user can give you doesn't join to the run that failed. This happens in four predictable places.

Streaming. The client gets a response ID at the end of the stream. If the server crashes mid-stream, there is no response ID, but there is a request ID and a partially rendered message. Generate the run ID before the first token and send it in the first SSE event, not the last. The client should store it with the message so a support agent can find a conversation that never completed.

Queues. A job payload should carry trace_id and run_id explicitly. Relying on the worker to look it up from a database row works until the row is written after enqueueing, which is exactly the case during a partial failure. Pass both.

Retries. A retry with a new run_id is correct for billing and telemetry, but it destroys the story. Model retries as child runs with parent_run_id, and show them collapsed under the original in the support view. When support says "it answered twice," you want to see one user turn with two attempts, not two unrelated rows.

Fallbacks. If a cheap model fails and a stronger model answers, that is a single user-visible outcome with two model calls. Same rule: one run, multiple model_call events.

A small amount of discipline at the boundary prevents months of confusion:

import uuid
from contextvars import ContextVar

run_ctx: ContextVar[dict] = ContextVar("ai_run")

def new_run(*, conversation_id: str, tenant_id: str, user_id: str | None) -> dict:
    ctx = {
        "run_id": str(uuid.uuid4()),
        "trace_id": format_trace_id(),
        "conversation_id": conversation_id,
        "tenant_id": tenant_id,
        "user_id": user_id,
    }
    run_ctx.set(ctx)
    return ctx

def event(kind: str, **payload) -> None:
    ctx = run_ctx.get()
    writer.enqueue({
        "run_id": ctx["run_id"],
        "trace_id": ctx["trace_id"],
        "kind": kind,
        "payload": redact(payload),
    })

The redact call is doing real work, and it deserves its own section.

Redact for the reader, not for the log

AI runs contain more sensitive material than typical application logs: full user messages, retrieved internal documents, tool arguments that include account numbers, and model output that may quote any of the above.

The instinct is to redact aggressively, which produces a support view full of [REDACTED] and no answers. The better approach is to redact by role: support needs to see enough to classify the failure, and engineering needs more, under stricter access.

A practical three-tier model:

• Tier 1 (all support): message lengths, language, tool names, tool status, error codes, retrieval counts, model name, prompt version, latency, and a truncated preview of the assistant's output (first 200 characters). No account numbers, no document bodies.
• Tier 2 (support lead, audited): full user message and full assistant output, with regex-detected emails, phone numbers, card-like digit strings, and national IDs masked. Retrieval results shown as titles and IDs only.
• Tier 3 (engineer, audited, time-boxed): full assembled prompt and full retrieved chunks, with access granted per ticket for 24 hours rather than permanently.

Implementation details that matter:

• Mask at write time for tier 1 previews, but store the unmasked payload encrypted with a key that tier 3 access can unwrap. If you only store masked data, you cannot debug a bug that depends on the actual text.
• Redact after you compute hashes. A hash of the full input is often enough to detect "same input as the failing case" without exposing content.
• Never log API keys, bearer tokens, cookies, or Authorization headers. Scrub them at the HTTP client layer so no future code path can forget.
• Treat retrieved document text as customer data. It usually is.

🚨 Production warning: an AI debug view is a data exfiltration surface with a nice UI. Anyone with support access can paste a customer's conversation into it and read back retrieved internal documents. Put the tool behind SSO, log every view, and scope it so a support agent can only open runs for tenants they are assigned to.

Record the tool contract, not just the call

Support tickets about agents are rarely about text. They're about actions: a refund that didn't happen, a ticket that was closed, an email that went to the wrong address. To debug those, you need to know what the tool promised as well as what it did.

For each tool call, record:

• the tool name and version;
• the arguments the model produced, and the arguments after your validation and coercion;
• the authorization decision, including which policy allowed it;
• the idempotency key, if the tool is a write;
• the external system's identifier for the side effect (refund_id, ticket_id, message_id);
• the result, or the structured error;
• the latency, and whether it was retried.

The distinction between "arguments the model produced" and "arguments after validation" is the one people skip. If the model asked for amount: "25" and your validator coerced it to 25.00, that's fine. If the model asked for amount: 2500 and your validator silently divided by 100 because that's what the old API expected, that's a bug you will only find by keeping both.

A structured tool result is worth more than a log line. Compare:

# Hard to debug: the model gets a string, support gets nothing
return f"Error: could not process refund for order {order_id}"

# Debuggable: the model gets a usable message, the system gets facts
return ToolResult(
    ok=False,
    code="payment_provider_timeout",
    retryable=True,
    user_message="The payment provider didn't respond. Nothing was charged.",
    details={"order_id": order_id, "provider": "stripe", "attempt": 2},
)

The ok, code, and retryable fields are what let your support view say "the refund was never submitted" instead of "the assistant said something wrong." They also let the agent decide whether to retry, which is the difference between a recoverable failure and a loop.

One more detail: record the tool definition hash in the run. When someone edits a tool description to "make the model behave better," you want to be able to say whether a ticket predates that change. Tool descriptions are prompt engineering by another name, and they need the same version discipline as prompts.

Give support safe actions, not credentials

Read-only debugging covers most tickets but not all. There are a few actions that let support resolve issues without engineering, and each one needs a deliberately narrow design.

Replay with the recorded context. Re-run the same prompt, same retrieved documents, same model snapshot, against a sandbox that cannot commit side effects. This answers "was it a fluke or is it reproducible?" without asking the user to try again. Sandbox mode must be enforced by the tool layer, not by a flag the model can influence.

Regenerate with a fix. After a retrieval miss is confirmed and the document is indexed, regenerate the answer for the same turn so the user sees a corrected response. This is a product feature, not a debug tool, but support will ask for it, so decide early whether it exists.

Disable the feature for one tenant. A per-tenant kill switch that degrades the UI to the non-AI path. Support leads should be able to pull it during a bad rollout without waiting for a deploy. Pair it with an automatic notification to the owning team so it isn't used silently.

Escalate with a bundle. One button that packages the run, its events, the versions, and the user's description into a ticket in the engineering queue, with the trace link attached.

What support should not have: database access, the provider dashboard, the ability to edit prompts, or the ability to run arbitrary tools as the customer. Every one of those turns a support agent into an unaudited production operator.

Design the escalation bundle before you need it

The handoff between support and engineering is where most time is lost. A good bundle is a single JSON or a zip with a stable shape, generated by the support view, containing:

• run row and all events, ordered;
• full prompt with secrets already scrubbed;
• retrieved documents as IDs plus the text that was actually in context;
• tool calls with arguments and results;
• versions, flags, and release;
• the user's own words, unedited;
• the support agent's classification and what they already tried.

The last two items are the ones engineers forget to ask for, and they're often decisive. "Customer says the total is wrong" plus "support reproduced with a different order and got the same wrong total" is a much better bug report than a trace alone.

Keep a template. If every escalation is a different shape, the on-call engineer spends the first ten minutes normalizing instead of investigating.

Dashboards answer "how many," case files answer "why"

Aggregate dashboards are necessary and frequently mistaken for debuggability. A chart showing that refusal rate doubled at 14:05 tells you something changed. It doesn't tell you which customers were affected, what they asked, or what the model was looking at.

The two surfaces serve different questions:

Question	Surface	Typical user
Is the feature healthy right now?	Metrics, SLOs, alerts	On-call engineer
Did this specific user get a bad answer?	Run timeline	Support
Is this failure new or has it always happened?	Cohort query over runs	Product / eng
Are users rejecting answers more this week?	Feedback rates by prompt version	Product

Build the timeline first. Dashboards without a case file produce alerts nobody can act on; case files without dashboards produce support tickets that never reveal a pattern.

Turn user feedback into a join key

Thumbs up and down are nearly worthless as sentiment and very valuable as pointers. Store feedback as an event on the run, with the run ID, the surface, and whatever the user typed. Then make the feedback rate a first-class metric sliced by prompt version, model, and retrieval index version.

The pattern that works: a thumbs-down opens a small form with four choices — "wrong information," "didn't answer my question," "did something wrong," "unhelpful tone." Those map cleanly onto the failure categories support already uses, and they let you rank prompt versions by bad-answer rate rather than by engagement.

Also capture implicit rejection: the user retyping the same question within thirty seconds, editing their message, or immediately asking a human. These signals are noisier but far more common than explicit feedback, and they're the ones that will show a regression on the day it ships.

The ten-minute reproduction test

Before launch, run this exercise with someone from support who has never seen the feature. Give them a ticket with a user email and a timestamp and nothing else. Start a timer.

They should be able to:

1. find the conversation (under two minutes);
2. identify the run that produced the bad answer, including fallbacks (under three);
3. read the timeline and state what the model was told and what tools ran (under three);
4. pick a failure category from your list (under one);
5. either resolve it or produce an escalation bundle (under one).

If they can't, the gap they hit is your next ticket. It's usually one of: no way to search by email, no run ID on the message, no tool results recorded, or a UI that shows JSON instead of labels.

This test is cheap, it uses no ML knowledge, and it catches the exact failure mode that makes AI features expensive to operate: every ambiguous ticket becomes an engineering interrupt.

Anti-patterns that look like debugging but aren't

"Check the provider dashboard." It shows requests and tokens. It does not show your retrieval, your prompt assembly, your tool authorization, or what the user saw. It's useful for billing disputes and nothing else.

"Ask the user to send a screenshot." Screenshots don't include the conversation ID, the model, or whether the answer was streamed and truncated. Put a copyable reference in the message footer instead.

"We log everything to Datadog." Then support needs a query language, index names, and permission to a system they don't use daily. Logs are for engineers; the support view is for support.

"The prompt is in the code, so we know what it was." You know what it is now. You don't know what it was at 09:14 last Tuesday, which is when the ticket was filed. Version prompts and record the version per run.

"Engineering will just look it up." Engineering will look it up for the first twenty tickets and then start deprioritizing them, which is rational, because each one costs forty minutes of a senior engineer's time to answer a question a support agent could have answered with a screen.

A pre-launch checklist for support debuggability

• Every assistant message carries a visible, copyable reference ID tied to a run.
• Runs are searchable by email, conversation, tenant, and time window.
• Each run stores prompt key and version, model and snapshot, index version, flags, and app release.
• Tool calls store arguments before and after validation, structured results, and external side-effect IDs.
• Fallbacks and retries are modeled as child runs, not independent runs.
• A support-facing timeline exists, with derived failure labels rather than raw JSON.
• Redaction is tiered, and tier 3 access is audited and time-boxed.
• Support can replay in a sandbox that cannot write.
• A per-tenant kill switch exists and is documented in the runbook.
• One-click escalation produces a consistent bundle.
• Feedback and implicit rejection events join to runs.
• A support agent who has never seen the feature has passed the ten-minute test.

The last item is the only one that proves the other eleven work.

What to build first if you have one sprint

If you can't do all of this, do three things in this order. Put a run ID on every assistant message and make it searchable by email. Write the event log with kind, seq, and payload, even if the payloads are thin at first. Build the timeline page with four derived labels: no retrieval, tool error, loop detected, and fallback model.

That combination turns "the AI is broken" into "the refund tool timed out at 14:02 and the model answered anyway." The second sentence is a bug with an owner. The first is a mood.

The model will get better on its own schedule. The debugging surface won't improve unless someone builds it, and the team that builds it early spends far less time answering the same question from support every week.