Instrumenting AI tool calls for incident review

The agent’s final answer looks reasonable. The customer says the wrong account was updated. The model trace shows a tool call named update_profile, but the log does not show which account, which fields, what authorization decision was made, or whether the provider retried the call. The incident review has a transcript and no reliable explanation.

Tool calls are where an AI workflow crosses from generated language into application behavior. Instrumenting only the model request misses the part that changes records, sends messages, charges money, or exposes data. The goal is to make each tool invocation reconstructible without copying every prompt and result into a broad-access log store.

This article describes a practical event model for tool calls: how to capture proposal, policy, execution, and outcome; how to link retries and queue jobs; how to protect sensitive arguments; and how to make the traces useful during an incident rather than merely detailed.

A tool call has more than one event

An agent framework may show one function_call span. For operations, that single span often hides several distinct decisions:

The model proposed a tool and arguments.
The application parsed and validated the arguments.
Authorization and policy allowed or denied the action.
The tool implementation called an internal or external service.
The service returned success, failure, or an ambiguous outcome.
The result was persisted, retried, compensated, or shown to a human.

Separate these events in your instrumentation. The model’s proposal is not the execution. A policy denial is not a tool failure. A timeout is not proof that the external service did nothing. If the incident dashboard has one field called tool_status, it cannot distinguish these states.

OpenAI’s Agents SDK tracing documentation is one concrete example of nested trace coverage: it records model generations, function calls, guardrails, handoffs, and custom events as spans. That is useful for seeing the workflow path, but applications still need their own business-level operation records and access controls. (OpenAI Agents SDK, tracing)

Give every logical action a stable operation ID

The request ID from a provider identifies one API call. The tool call ID from a model identifies one generated call. Neither necessarily identifies the business operation across retries, queue redeliveries, or human approval.

Create an operation_id for the logical side effect, and attach it to:

the agent task and trace;
the tool proposal and policy decision;
each network attempt;
the downstream idempotency key where supported;
the durable operation record;
any approval, compensation, or webhook event.

Each attempt should also have its own attempt_id. This lets you group the attempts while preserving their individual timing and errors. Use opaque IDs; do not encode an email address, customer name, or payment details in the trace ID.

For one user request, there may be several tool operations. Updating a record and sending a notification are separate operations, even if they share a task ID. The operation ID provides the join; the trace ID provides the execution tree.

Capture the proposal, but do not blindly log raw arguments

The proposal is important evidence: which tool did the model choose, and what values did it supply? But raw arguments can contain personal information, credentials, or large text fields. Logging everything into an analytics backend creates a new data exposure.

For each tool, define a logging schema. Example for an order update:

{
  "event_name": "ai.tool.proposed",
  "operation_id": "op_83a2",
  "tool_name": "update_shipping_address",
  "target_ref": "order_ref_6d12",
  "argument_fields": ["street", "city", "postal_code"],
  "argument_hash": "sha256:...",
  "prompt_version": "support-agent-v12",
  "tool_schema_version": "orders-tools-v4"
}

The ordinary trace can retain the target reference, field names, and a canonical hash. A restricted audit store can retain a redacted or access-controlled parameter record if the investigation requires it. Avoid sending addresses, payment tokens, or entire documents to every trace destination.

Do not trust a blocklist redactor alone. Tool arguments are structured, so use an allowlist of fields that are safe to export. If a new field is added to a schema, it should not automatically start appearing in logs without review.

Record authorization as its own event

For every tool operation, record the decision made by the application:

{
  "event_name": "ai.tool.authorization",
  "operation_id": "op_83a2",
  "actor_ref": "user_opaque_17",
  "tenant_ref": "tenant_42",
  "action": "address:update",
  "target_ref": "order_ref_6d12",
  "decision": "denied",
  "reason_code": "user_scope_mismatch",
  "policy_version": "orders-policy-v9"
}

This event should come from the authorization boundary, not from the model. Do not log the full policy or sensitive business context unless it is required and protected. Stable reason codes make it possible to aggregate denied calls without indexing raw exception text.

If a tool is allowed, log which actor’s authority applied: the requesting user, a service identity, or an approver. In multi-tenant systems, record a tenant reference and verify that it matches the resource scope. A broad service token can make a tool call succeed while violating the user’s actual permissions; the trace should make that visible.

Separate execution status from response status

The model may report success even if the tool failed, and a tool can succeed even if the application loses the response. Record the downstream status separately from the assistant’s final response.

Useful execution statuses include:

not_started;
denied;
in_progress;
succeeded;
failed;
unknown;
compensated.

unknown matters. A connection can time out after the provider commits. If the trace records only error, a later worker may repeat the action. Mark the operation unknown, query the source of truth, and reconcile before retrying.

For every side-effecting tool, capture a downstream request ID or object reference if available. Record a safe result summary and the resource version before and after the operation. Do not treat a 200 response as proof that the business goal succeeded; validate the returned state.

Make retries visible as attempts under one operation

An agent tool may be retried by the model, an SDK, your application, a queue, or a workflow engine. If each layer emits unrelated logs, incident review cannot tell whether the same operation was sent twice.

For every attempt, record:

operation and attempt IDs;
attempt number and retry owner;
normalized request hash;
error class and status code;
Retry-After or backoff delay where applicable;
whether cancellation was requested;
estimated or actual cost if the downstream call is metered;
provider request ID and final outcome.

The same logical operation should reuse its idempotency key and exact parameters when the downstream service supports idempotency. If the model proposes different parameters on a retry, treat that as a changed proposal, not a transparent retry. Require a new operation or renewed approval.

Avoid logging high-cardinality identifiers as metric labels. Store operation IDs in logs and traces; use low-cardinality dimensions such as tool name, result class, model family, and policy outcome in metrics. This keeps dashboards usable and prevents an unbounded series count.

Propagate context across queues and handoffs

Tool calls often execute in a background worker or after a human approval. Propagate the operation ID and trace context through queue metadata, workflow state, and webhook handlers. The worker should create a child span or event under the original operation rather than creating an unrelated trace.

For multi-agent systems, record the handoff reason and destination as controlled fields. A specialist agent’s tool call should remain linked to the original user request, but the access boundary should still be rechecked at the destination. Context propagation is for observability; it is not a permission token.

Clock timestamps alone are not enough to reconstruct causal order. Distributed hosts can have skew, and webhook events may arrive out of order. Use parent span IDs, sequence numbers, operation state transitions, and provider event IDs to establish the actual sequence.

Redact before data leaves the application boundary

OpenAI Agents SDK documentation notes that generation spans and function spans can capture inputs and outputs, and that sensitive-data capture is enabled by default in the SDK unless configured otherwise. It also documents that tracing is unavailable for organizations using OpenAI APIs under Zero Data Retention. These are provider-specific defaults that should be checked when selecting an exporter and deployment configuration. (OpenAI Agents SDK, sensitive data)

Redaction must happen before every export. The same SDK documentation warns that adding a redaction processor alongside a default exporter does not guarantee the default exporter will receive only redacted data if the redactor fails. If redaction is a hard requirement, own the export chain, fail closed when redaction fails, and test that no other processor has already received the original payload.

The safest approach is often to construct a safe event schema at the tool boundary, rather than serializing arbitrary tool objects and attempting to scrub them later. Keep a restricted audit record separate from general telemetry, and define retention for both.

A tool instrumentation wrapper

The wrapper below is illustrative. It shows where events belong, not a specific tracing SDK API:

def invoke_tool(operation, tool, args, principal, audit, tracer):
    with tracer.start_span("ai.tool.execute") as span:
        span.set_attribute("ai.tool.name", tool.name)
        span.set_attribute("app.operation_id", operation.id)
        span.set_attribute("app.prompt_version", operation.prompt_version)

        decision = authorize(principal, tool.name, args)
        audit.record_authorization(operation, decision.safe_record())
        if not decision.allowed:
            span.set_attribute("app.policy.outcome", "denied")
            operation.mark_denied(decision.reason_code)
            return ToolResult.denied(decision.reason_code)

        operation.mark_in_progress()
        try:
            result = tool.execute(args, idempotency_key=operation.idempotency_key)
        except TimeoutError:
            operation.mark_unknown()
            span.set_attribute("app.tool.outcome", "unknown")
            raise
        except ToolError as exc:
            operation.mark_failed(exc.safe_code)
            span.set_attribute("app.tool.outcome", "failed")
            raise
        else:
            operation.mark_succeeded(result.safe_reference())
            span.set_attribute("app.tool.outcome", "succeeded")
            return result

The wrapper logs a safe reason code and result reference, not arbitrary exception text or raw arguments. In a real implementation, authorization must use the validated target and authenticated principal; tool.execute must be bounded and idempotent for writes; and state changes must be persisted durably.

Build an incident query path

Instrumentation is only useful if responders can find the events quickly. Support and on-call should be able to search by a safe task reference, then navigate to:

the user-visible request and operation state;
the model turn and prompt version;
the proposed tool call;
the authorization decision;
the execution attempt and downstream request ID;
any retry, compensation, or human approval;
the final business object state.

Do not make the responder query an unrestricted log warehouse to answer one customer’s question. Provide a restricted case view that applies the same tenant and role rules as the application. If raw arguments are necessary, require a stronger audit path and record who accessed them.

Test the instrumentation, not just the tool

Add tests that verify:

every tool attempt has an operation and attempt ID;
denial events do not contain raw sensitive content;
a timeout produces unknown, not failed or succeeded;
retries share one operation ID and key but have distinct attempt IDs;
a successful write records a downstream reference;
a policy version and tool schema are present;
redaction failure prevents export to all destinations;
queue handoffs preserve correlation IDs;
metrics do not include high-cardinality user or operation IDs.

Test the outage path too. If the tracing backend is unavailable, the business operation should follow a documented policy; if audit persistence is mandatory for a write, the application may need to block the write until the audit event is durably recorded. Observability and audit have different availability requirements, so make that distinction explicit.

Metrics that help without replacing traces

Track tool call count, latency, error class, policy decisions, retry ratio, and unknown outcomes by tool and action type. For workflows with human review, measure approval, edit, rejection, and time-to-decision. For write operations, measure compensation rate and side effects per logical operation.

Metrics identify patterns; traces explain individual cases. A spike in unknown outcomes on update_shipping_address tells you where to look. The trace for one operation shows whether the provider timed out after committing or whether the request never reached it.

Avoid “model tool accuracy” as a single metric. It conflates tool selection, argument correctness, authorization, execution success, and the business outcome. Keep these as separate dimensions so the team can fix the stage that failed.

Keep traces and audit events on different retention paths

Sampling is reasonable for high-volume diagnostic traces, but a required audit event should not disappear because the trace sampler chose not to keep the request. A trace is a debugging view and may be incomplete; an audit record documents a consequential action and needs the durability and access control your policy requires.

Use separate pipelines or clearly separated data classes:

general metrics for aggregate trends;
sampled traces for performance and model behavior;
protected audit events for writes, approvals, and denials;
restricted raw content only when a specific diagnostic or legal purpose requires it.

Define retention independently. A trace may expire after the support window, while a financial audit record may need longer retention. Do not store full prompts in an audit table merely because it is durable. Keep the minimal record of actor, action, target, policy version, decision, and result.

Sampling should also be biased toward useful cases. Keep a representative sample of successful calls, all high-risk denials or writes where required, and traces triggered by error or latency thresholds. Record whether a trace was sampled so analysts do not mistake the retained set for a uniform sample of traffic.

Use controlled fields instead of high-cardinality labels

Instrumentation can create its own reliability problem if every user, order, or operation ID becomes a metric label. High-cardinality time series grow quickly and can make dashboards expensive or unusable.

Put stable low-cardinality values in metrics: tool name, result class, policy decision, model family, and workflow type. Put opaque identifiers in logs and traces where investigators can search them. Avoid using free-form exception messages or model-generated tool names as metric dimensions; normalize them into controlled categories.

Version fields matter too. Include prompt, tool-schema, and policy versions in the trace or event record so a regression can be correlated with a rollout. In metrics, use a bounded release label or compare versions in a controlled dashboard rather than keeping every content hash as a permanent series dimension.

Walk one incident from proposal to business state

Suppose a user asks an agent to update an address. The model proposes update_shipping_address with an order reference. The policy check denies the call because the order belongs to a different tenant. The assistant then tells the user it cannot access that order.

A useful trace shows the user operation ID, the prompt and tool-schema versions, the proposed tool and a safe target reference, the tenant_mismatch policy decision, and the fact that the tool was not executed. The audit record links to the denial. The model’s final text is associated with the same operation but is not treated as proof of the authorization outcome.

Now consider the opposite: authorization allows the call, the provider times out, and the browser retries. The first attempt is unknown; the second reuses the same operation key. Reconciliation finds that the address was already changed, so the application returns the stored result instead of applying it again. The trace makes the retry visible, while the operation record makes the business effect idempotent.

These two cases illustrate why “log the model response” is inadequate. In one case, the application denied a validly formed proposal. In the other, the external side effect may have succeeded despite a missing response. The incident path depends on the tool lifecycle, not only on the generated words.

Normalize provider and framework events

Agent SDKs and observability platforms use different names for model turns, tool calls, handoffs, and guardrails. Adopt a small application-owned event vocabulary and map provider-specific spans into it. Keep the original provider request ID and event type as metadata for deep debugging, but make cross-provider dashboards use stable concepts such as tool.proposed, tool.authorized, tool.executed, and tool.reconciled.

OpenTelemetry’s GenAI semantic conventions are maintained in a dedicated repository and continue to evolve. Check their current status before treating an attribute name as permanent. (OpenTelemetry GenAI semantic conventions) A vendor trace UI can be convenient, but the application should retain enough stable identifiers and business events to reconstruct an incident if the framework or exporter changes.

Do not make a prompt or tool contract depend on a trace-only field. Tracing may be disabled for privacy, ZDR, or performance reasons. The operation ID and authorization decision still need to exist in the application’s own workflow state.

During an incident review, use the normalized event sequence to answer three separate questions: what did the model request, what did the application permit, and what changed in the external system? If any one of those is inferred from the final answer, the instrumentation is incomplete.

Version the event schema and keep consumers tolerant of optional fields. When a tool adds a new argument, do not let an old dashboard silently expose it; review the field’s sensitivity before adding it to an exporter or support view.

A practical checklist

Before enabling a tool in production, verify:

The tool has a stable name and versioned schema.
Proposals, authorization, execution, and outcomes are separate events.
Every logical write has a stable operation ID and idempotency strategy.
Every attempt has its own ID and safe error classification.
Unknown outcomes enter reconciliation, not blind retry.
Arguments and results are minimized and redacted before export.
Trace context survives queues, handoffs, and webhooks.
Audit events are durable where the action requires them.
Support can locate a safe case record without broad log access.
Instrumentation tests cover redaction failures and duplicate delivery.

The best tool-call trace is not the one with the most raw content. It is the one that proves what the model proposed, what policy allowed, what the service did, and what state resulted—while keeping sensitive inputs inside the boundary that is authorized to see them.