A team upgrades from one model to a newer, more capable one. The demo is better. The benchmark scores are better. Two weeks later, refund disputes are up, and nobody can say why, because the old runs were never recorded in enough detail to compare. The prompts were edited in place. Tool arguments came from a provider log that has since rotated. The approval for the disputed refunds exists as a Slack message.
So the team does what teams do in that situation: they argue about models. Someone says the new one is "more aggressive." Someone else says the prompt is fine. The discussion is unfalsifiable because there is no evidence, and the only way to get evidence is to wait for more incidents.
The model was probably not the problem. The problem was that the system had no memory of its own decisions, so the first controlled experiment happened in production, with customers as the control group.
That's the argument for an audit log: it is the instrument that turns "the agent feels worse" into a testable claim. It is also the thing regulators ask for, the thing incident reviews need, and the thing that makes an evaluation set possible. A better model without an audit log is an improvement you cannot measure, defend, or roll back with confidence.
The upgrade you cannot evaluate
Model upgrades are the most common change to an AI product and the least rigorously tested. There are practical reasons: outputs are non-deterministic, the new model may not accept the same parameters, and the prompt that worked may need adjustment. But the deeper reason is usually missing data.
To evaluate an upgrade on your own workload you need, at minimum:
- a set of real historical inputs;
- the context each one was given, including retrieved documents;
- the output that was actually produced and whether it was accepted;
- the tool calls that followed and whether they succeeded;
- the version of the prompt, tools, and retrieval index at the time.
Only the first item is easy. The rest is exactly what an audit log records. If you have it, the upgrade becomes a replay experiment: run the same inputs and context against the new model, compare outputs against the old ones, and route disagreements to human review. If you don't, the upgrade becomes a vibe check on fifty hand-written examples, which is how you end up shipping a regression in a category nobody thought to test.
💡 Practical note: "we'll build the eval set after the audit log" is the correct order, not a postponement. An eval set built from memory and imagination tests the failure modes you already fear. An eval set built from logs tests the ones your users actually hit.
Three jobs an audit log does that a better model cannot
It's worth being precise about what an audit log is for, because "logging" is a phrase that gets nodded at and then implemented as logger.info(response).
Reconstruction. Given an outcome — a refund, a closure, a wrong answer, a customer complaint — you can rebuild the chain of events that produced it. Not the model's internal reasoning, which is unavailable anyway, but the observable inputs, decisions, and actions. Reconstruction is the difference between "the agent did something wrong" and "the agent called apply_credit with amount_cents=25000 because the retrieval returned a policy page for the annual plan."
Accountability. Every decision has an actor: the model, a policy, an approver, or a human editor. An audit record names them. When the question is "who authorized this," the answer should not be "the agent," which is a category error.
Evidence. For evaluation, for incident review, and for anyone external who has a legitimate right to ask. Without evidence, the only defensible position is "trust us," and that position gets more expensive as the feature gets more autonomous.
A better model improves the distribution of outcomes. An audit log makes the distribution visible. You need the second one to know whether you got the first.
Logs, traces, and audit records are different artifacts
These three get conflated because they all contain timestamps and JSON. They serve different consumers and have different retention, access, and integrity requirements.
| Property | Application log | Trace | Audit record |
|---|---|---|---|
| Primary consumer | On-call engineer | Performance debugging | Compliance, legal, incident review |
| Retention | Days to weeks | Days to weeks | Months to years |
| Mutability | Rotates, sampled | Sampled, dropped | Append-only, tamper-evident |
| Content | Errors, warnings, context | Spans, timings, attributes | Decisions, actors, approvals, side effects |
| Access | Engineering | Engineering | Audited, role-restricted |
| Completeness | Best effort | Sampled | Every consequential action |
| Key question | "What broke?" | "Where did time go?" | "Who decided what, and can we prove it?" |
You will have all three, and they should share identifiers. The trace ID belongs in the audit record so an investigator can jump from "refund approved at 14:02" to the spans that show the retrieval took 3.4 seconds. But they should not be the same store, because the retention and access rules are incompatible. A 400-day audit record with full customer text needs different encryption and access controls than a seven-day trace with sampled attributes.
The mistake to avoid is treating the trace as the audit log. Traces are sampled, they're lossy by design, and their backends are optimized for short retention. An audit trail that disappears after two weeks is a debugging aid wearing a compliance costume.
What belongs in an agent audit record
An audit record for an agent needs to answer a small set of questions per event: who, what, when, with what authority, with what result, and with what consequence in the outside world.
A workable event shape:
{
"event_id": "01JQ8Z3K9V2N7R4T6B1C0XW5YA",
"occurred_at": "2026-03-04T09:12:44.118Z",
"run_id": "8f2c1e40-6b5a-4a01-9a3f-2d1b7c9e4f10",
"conversation_id": "c_9f31",
"actor": {"type": "agent", "id": "support_triage", "version": "2026.02.3"},
"on_behalf_of": {"type": "user", "id": "u_9182", "tenant_id": "t_44"},
"action": {
"type": "tool_call",
"tool": "create_refund",
"args_hash": "sha256:5b1e9c...",
"args_redacted": {"order_id": "A-1182", "amount_cents": 2500, "reason": "damaged_item"}
},
"authorization": {"decision": "allow", "policy": "refund_policy_v7", "grant_id": "g_771"},
"approval": {"required": true, "approved_by": "u_3311", "approved_at": "2026-03-04T09:12:39Z"},
"outcome": {"status": "success", "external_id": "re_3PkQ", "latency_ms": 412},
"versions": {"prompt": "refund_agent@14", "model": "provider/model-2026-01", "index": "kb_2026_03_01"},
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}
Every field earns its place, but a few are routinely omitted and painfully missed later.
on_behalf_of. Without it, you cannot answer "which customers were affected" — the single most important question in an incident review. An agent identity alone doesn't scope the blast radius.
args_hash plus redacted args. The hash lets you prove that two events used the same arguments without storing sensitive values in a widely readable store. The redacted copy makes the record legible. If you store only the hash, investigators can't read it. If you store only the raw args, your audit log becomes a PII repository.
authorization. The decision, the policy version, and the grant. When a permission bug surfaces six months later, this is how you determine which calls were affected. It also distinguishes "the model chose badly" from "the policy allowed it," which are different bugs with different owners.
approval. Human decisions are the most important events in the log and the ones most often stored somewhere else — a ticketing system, an email, a Slack thread. If the approval isn't in the audit record, the record says a refund happened and doesn't say who said yes.
outcome.external_id. The identifier from the downstream system. This is your only reliable link between "the agent did a thing" and "the thing exists in Stripe, Zendesk, or the warehouse." Without it, reconciliation is manual.
versions. Prompt, model, index, and the tool definition hash. A version field that only names the model is nearly useless, because most behavioral changes come from prompts and tools.
Append-only, and preferably tamper-evident
An audit log that can be updated is a log that can be made to say anything. If your audit table allows UPDATE, then the first time a bug corrupts a record, someone will "fix" it, and the fix will remove the evidence.
At minimum, enforce append-only at the database level:
create table audit_events (
seq bigserial primary key,
event_id text not null unique,
occurred_at timestamptz not null,
run_id uuid not null,
tenant_id uuid not null,
actor_type text not null,
actor_id text not null,
action_type text not null,
payload jsonb not null,
prev_hash text not null,
hash text not null
);
create rule audit_events_no_update as on update to audit_events do instead nothing;
create rule audit_events_no_delete as on delete to audit_events do instead nothing;
revoke update, delete on audit_events from app_role;
grant insert, select on audit_events to app_role;
The hash chain is the next step. Each record includes the hash of the previous record plus its own content, so removing or altering an event breaks the chain from that point forward.
def record(conn, event: dict) -> None:
prev = conn.execute(
"select hash from audit_events order by seq desc limit 1 for update"
).fetchone()
prev_hash = prev["hash"] if prev else "genesis"
body = canonical_json({**event, "prev_hash": prev_hash})
digest = hashlib.sha256(body.encode()).hexdigest()
conn.execute(
"""insert into audit_events
(event_id, occurred_at, run_id, tenant_id, actor_type, actor_id,
action_type, payload, prev_hash, hash)
values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(event["event_id"], event["occurred_at"], event["run_id"], event["tenant_id"],
event["actor"]["type"], event["actor"]["id"], event["action"]["type"],
json.dumps(event), prev_hash, digest),
)
A single serialized writer is a bottleneck at high volume, so most production systems relax this to per-tenant chains or periodic Merkle roots over batches. Both preserve the property that matters: you can prove a record wasn't silently changed. Choose based on your write rate, not on aesthetics.
🚨 Production warning: don't put raw prompts, tool results, or retrieved documents in the audit table. Store hashes and references to an encrypted content store with its own retention. The audit log should be readable by auditors and useless to an attacker who gets a read-only dump.
Reconstructing one decision end to end
Here's what the payoff looks like. A customer disputes a $25 credit issued on their account and says they never asked for it.
With the record above, the investigation takes minutes:
- Find the event by
external_id = re_3PkQ, or by customer and date range. - Read
on_behalf_of— the run was for user 9182 in tenant 44, which matches the complainant. - Read the conversation events leading up to it. The user wrote "just give me the twenty-five back and I'll drop it," which is ambiguous but arguably a request.
- Read the retrieval event. The policy page returned was for the annual plan, but the customer is on monthly, where the damage-credit limit is $10.
- Read the authorization event. The policy check passed because it read
amount_cents <= 2500from the annual-plan clause it retrieved. - Read the approval. An approver approved it in 11 seconds, likely without reading the retrieved policy.
- Conclusion: the model applied the wrong policy clause because retrieval returned the wrong document, and the approval step didn't catch it.
That's a chain of five findings, each with an owner: retrieval (wrong document), prompt (didn't verify plan type), policy tool (no plan-aware limit), approval UX (too easy to rubber-stamp). None of it is available from a model score.
Now compare with the common alternative: a log line that says refund created: 2500 for order A-1182. The investigation ends with "we're not sure, but the model may have misunderstood."
Retention, erasure, and the tension nobody enjoys
Audit logs and privacy rules pull in opposite directions. Audit records want to be complete and long-lived; data protection rules want collection to be minimal and deletion to be prompt. Both are legitimate, and pretending the tension doesn't exist produces either an audit log that's legally risky or a retention policy that destroys evidence.
Some practical resolutions:
Separate identity from content. The audit record can retain "user 9182 was refunded 2500 cents at this time under policy v7" for years. The conversation text that produced it can have a 90-day retention with the audit record holding only a hash. You keep accountability without keeping the transcript.
Distinguish audit obligations from product data. If a legal or regulatory obligation requires retaining the record, that's a documented basis, not an accident of your logging pipeline. Write it down, and make the retention job encode it explicitly rather than relying on "we never delete anything."
Design deletion as a scoped operation. When a user exercises a deletion right, you need to remove their content without breaking the integrity of the audit chain. Hash-chained designs handle this by keeping the event and replacing content references with tombstones — the chain still verifies, the content is gone. Plan for it before the first request arrives, because retrofitting is expensive.
Make retention configurable per event class. permission_decision and approval may need seven years. model_call with token counts may need one. debug_payload may need seven days. One global retention number is either wasteful or non-compliant, usually both.
For context on why this matters to someone other than your legal team: the EU AI Act requires high-risk systems to technically allow automatic recording of events over their lifetime (Article 12), and providers to keep those logs for a period appropriate to the intended purpose, at least six months (Article 19). HIPAA's Security Rule requires audit controls for systems containing ePHI. SOC 2 expects monitoring and evidence of access decisions. You may not be in scope for any of these, and I'm not your compliance officer. But the direction of travel is consistent: if an automated system makes consequential decisions, someone will eventually ask for the record.
Mining the log for the eval set
The audit log's second life is as a source of test cases. A few queries turn history into evaluation material.
-- Low-quality outcomes with full context available
select r.run_id, r.prompt_version, r.model, e.event_id
from ai_runs r
join ai_run_events e on e.run_id = r.run_id and e.kind = 'user_feedback'
where r.started_at > now() - interval '90 days'
and e.payload->>'signal' = 'thumbs_down'
and e.payload->>'reason' in ('wrong_information', 'did_something_wrong')
order by random()
limit 200;
Then, for each sampled run, materialize the input, the retrieved context, the tool results, and the recorded output into a fixture. That fixture is the beginning of a regression suite that reflects real usage rather than imagined usage.
Three cautions.
Sample across outcomes, not just failures. A suite of 200 bad cases will tell you whether a new model fixes them and nothing about what it breaks. Include accepted answers, borderline ones, and cases where the agent correctly declined.
Record the expected outcome at the time of curation, not by trusting the old output. The old output is evidence of what happened, not a statement of what should happen. A human has to label it, or you're training the suite to preserve existing bugs.
Version the fixtures with the prompt and tool definitions they were captured under. A fixture that assumes a tool called get_order will silently stop testing anything when the tool is renamed, unless the mismatch fails the suite.
The upgrade protocol an audit log enables
With historical records, a model change becomes a repeatable procedure rather than a leap of faith.
- Freeze a baseline. Pick a date range, and export the inputs, context, and outcomes for a stratified sample across intents and outcomes.
- Replay against the candidate model. Same prompt, same retrieved context, same tool definitions. Only the model changes. This isolates the variable.
- Diff the outputs structurally. For structured outputs, compare fields. For free text, use a rubric-based grader plus a sample of human review. Record agreement rates, not just averages.
- Replay the tool trajectories. If the model chooses different tools or different arguments, that's often more consequential than different wording. Count trajectory divergence separately.
- Shadow in production. Route a small percentage of traffic to the candidate, with full audit records for both paths, and compare acceptance and escalation rates after a week.
- Decide with a stated threshold. For example: no more than a 1-point drop in task success, at least a 3-point gain in resolution rate, and no new failure category above 0.5% of traffic.
- Keep the old path available. Rollback is a flag flip if prompt and model versions are recorded per run — which they are, because the audit log required it.
Steps 1, 4, and 6 are only possible with records. This is why the audit log comes first: it's not a compliance tax, it's the experimental apparatus.
What an audit log will not tell you
Honesty about limits keeps you from over-trusting the record.
It won't tell you why the model produced a token sequence. Provider-side reasoning traces may help, but they're not a causal explanation, and they shouldn't be treated as ground truth.
It won't capture what wasn't logged. If the agent read a cached value that bypassed your event emitter, the record has a gap. Audit coverage is a property you have to test — pick a sample of runs and verify that every tool invocation has a corresponding event.
It won't help if events are written after the fact by a different code path than the one that acted. The event must be written at the boundary where the action happens, ideally in the same transaction as the side-effect record, or you'll have actions without audit entries and audit entries for actions that rolled back.
It won't fix a team that doesn't read it. An audit log without a review cadence — weekly sampling, incident triggers, periodic access reviews — is a very expensive archive.
The cost objection, priced
The usual reason audit logging gets deferred is storage cost. It's worth doing the arithmetic instead of guessing, because the number is usually smaller than the meeting assumes.
Take 50,000 agent runs a day, eight audit events per run, and an average event of 500 bytes of JSON after trimming payloads to identifiers, hashes, and decisions. That's 200 MB a day of raw JSON. Postgres TOAST compression and a realistic index overhead might triple the on-disk footprint, call it 600 MB a day, or roughly 220 GB a year. On commodity block storage that's a small monthly line item, and it's a rounding error next to the token spend of 50,000 runs.
What makes audit logs expensive isn't the rows. It's storing full prompts, retrieved documents, and tool results in the same table, indexing them for search, and replicating them to three environments. Separate the decision record from the content it references, keep the content store on object storage with lifecycle rules, and the cost stays proportionate.
There's a second, quieter cost: write latency. A synchronous insert on every tool call adds milliseconds and a failure mode. For read-only calls, buffering with a bounded queue and an at-least-once flush is fine. For consequential writes, the audit insert belongs in the same transaction as the side-effect record, and if it fails, the action should fail with it. That asymmetry is deliberate: a missing audit row for a read is a gap in observability, and a missing audit row for a refund is an unaccountable action.
Auditing the audit log
An audit log with no coverage measurement decays quietly. Two metrics keep it honest.
Coverage rate. For a sample of runs each week, count tool invocations observed in traces versus audit events emitted. A gap means some code path bypassed the emitter — usually a new SDK, a batch job, or a retry wrapper added by someone who didn't know the convention. Publish the number. Coverage that nobody watches drops to 80% within two quarters.
Chain verification. Run the hash-chain verifier nightly over the last N days. It should report zero breaks. When it reports one, you've either got a bug in the writer, a manual deletion, or a restore that lost rows — all of which you want to know about before an auditor does.
Add a third if you have humans in the loop: decision completeness, the share of consequential actions with a recorded approver. It should be 100% for the classes you designated as requiring approval. Anything less is a control that exists in policy and not in practice.
A phased plan that fits in a quarter
Weeks 1–2: define the event vocabulary. Five action types cover most systems: model_call, retrieval, tool_call, policy_decision, human_decision. Add outcome as an attribute rather than a type.
Weeks 3–4: emit at the boundaries. Wrap the HTTP client, the tool executor, and the policy check. Write the event in the same unit of work as the action where possible, and fail closed if the audit write fails for consequential actions.
Weeks 5–6: make it queryable. Index by tenant_id, occurred_at, run_id, and external_id. Build one view that reconstructs a run in order. This is the same view support needs, so build it once.
Weeks 7–8: retention and access. Classify event types, set retention per class, put the store behind SSO with audited access, and decide what gets hashed versus stored.
Weeks 9–10: mine it. Export a first evaluation set of 150–300 labeled cases, stratified by outcome and intent.
Weeks 11–12: run the first upgrade through the protocol. You'll learn more about your feature in that fortnight than in the previous two quarters of prompt tweaking.
The decision, stated plainly
If you have limited engineering time and a choice between "improve the model" and "record what the system does," record first. Model improvements are abundant, cheap, and reversible; they arrive on a schedule you don't control. Institutional memory is scarce, expensive, and irreplaceable.
A better model raises the ceiling. An audit log raises the floor, because it lets you find out when you're below it.
The test is simple. Pick a decision your agent made last month — a refund, a ticket closure, a recommendation. Try to reconstruct it from your current systems in under an hour, including who approved it and what context the model had. If you can, you already have the audit log, whether or not you call it that. If you can't, a better model won't help you answer the question next month either.