An agent is asked to onboard a new client. It creates the workspace, provisions forty seats, assigns the admin role, and then calls the billing API to purchase the seat pack. Billing returns 402 Payment Required. The agent reports failure and stops.
The workspace exists. Forty invitations are queued. No seats were purchased. For the next six hours, every invited user can sign in to a workspace with no license, and the sales team sees a deal marked "provisioned." Nobody wrote a line of code that does this; it is just what happens when a multi-step write stops in the middle.
This is the half-success problem, and it is the defining operational hazard of giving an agent write access. Model errors get the attention. Half-completed writes cause the incidents.
Why the transaction you want does not exist
In a single database, the answer is a transaction. Either all rows change or none do. The moment an agent calls a third-party API, that guarantee is gone. There is no distributed transaction across your database, Stripe, your identity provider, and the customer's Slack workspace. Two-phase commit exists, but essentially no SaaS product will enlist in it, and the operational cost of running it across vendors would be absurd even if they did.
The standard alternative is a saga: a sequence of local transactions where each step has a compensating action that semantically undoes it if a later step fails. The term comes from Garcia-Molina and Salem's 1987 work on long-lived transactions, and the core insight has not changed. You are not rolling back. You are writing a second, forward-moving action that reverses the effect of the first, and that action can fail on its own.
This distinction is the source of most production bugs in agent workflows. Teams write try: ... except: undo() and treat undo() as if it were a rollback. It is not. It is a new API call with its own latency, its own failure modes, and its own side effects.
Compensation is not rollback
Four differences matter, and each one changes the design.
It is not atomic. The original action is visible to other users, other services, and reporting jobs while the compensation is in flight. A customer may see the workspace, click the invite, and start working before it disappears. Compensation needs to be designed for that window, not around it.
It can fail. The refund API can time out. The delete endpoint can return 409 Conflict because the resource acquired a dependency. A compensation failure is a real incident, not an edge case, and it needs its own retry policy and alert.
It is observable. A refund appears on a customer's statement as a credit. A deleted record may persist in an audit log. Compensating a sent email requires sending another email. You cannot make the first one not have happened.
It may be impossible. Some actions have no inverse. You cannot un-send a webhook to a customer's endpoint, un-ship a package, or un-tell a user that their account was upgraded. For those steps, the only viable strategy is to delay them until nothing can fail afterward, or to require human confirmation.
🚨 Production warning: if your agent has a step with no compensating action, that step must be last, gated, or idempotent-by-design. There is no clever workaround.
Three shapes of half-success
Not every partial failure needs a saga. Distinguishing the shapes keeps you from over-engineering the easy cases and under-engineering the hard ones.
Shape 1: partial write inside one system. The agent creates an invoice and three line items, then fails on the fourth because a product id is invalid. Here a database transaction is available, and the right fix is to make the whole write one transaction. Do not build a saga for something your database can do atomically. The failure usually means the tool exposed too many small endpoints instead of one coherent operation.
Shape 2: committed, then failed downstream. The workspace is created, the seat purchase fails. This is the saga case. The first step committed in its own system, and no transaction spans the two.
Shape 3: unknown outcome. The request timed out. You do not know whether the charge succeeded. This is the worst shape, because both retrying and compensating are unsafe until you determine what happened. The only correct move is to query the downstream system for the state of the operation, using an idempotency key or a natural key, before doing anything else.
committed? ──yes──> run compensation (reverse order)
│
├──no───> safe to retry forward
│
└──unknown──> reconcile first; do not guess
A surprising number of "agent went rogue" reports are shape 3. The agent retried a POST after a timeout, the first request had actually succeeded, and the customer got two of something. The model did the reasonable thing. The tool did not give it a way to find out.
Try forward recovery before you undo
Compensation is the intuitive response to a failure: put things back. It is often the wrong one, because the goal is not a clean slate. The goal is a consistent state that matches what the user asked for.
If step 3 of 5 failed because of a transient error, the cheapest consistent state is usually "step 3 eventually succeeds." Undoing steps 1 and 2 throws away work that was correct, costs more API calls, and creates a window where the user sees a resource appear and disappear.
Decide with a simple rule:
| Situation | Prefer | Why |
|---|---|---|
| Transient failure, all prior steps valid | Forward retry | Preserves correct work; no user-visible churn |
| Permanent failure in a middle step | Compensate completed steps | The sequence cannot finish as designed |
| User cancelled | Compensate unless work is useful | Respect intent; stop spending |
| Policy or validation failure | Compensate, then stop | Retrying reproduces the same rejection |
| Unknown outcome | Reconcile, then decide | Guessing creates duplicates |
The concept that makes this concrete is a pivot point. Before the pivot, failure means undo. After the pivot, the run is considered committed and failure means finish forward, even if that requires retries and human help. For an onboarding flow, the pivot is usually the moment the customer is charged or the workspace becomes visible. Everything before it should be reversible; everything after it should be forward-only.
Choosing the pivot is a product decision, not a technical one. Put it as early as the business allows, because reversible work is cheaper to reason about.
Make steps compensable by design
The cheapest compensation is the one you never have to run, because the step was designed to be provisional.
Reserve instead of consume. Instead of "purchase 40 seats," do "reserve 40 seats for 15 minutes." A reservation expires on its own if the run fails. You still need a confirm step, but the failure path is a no-op rather than a refund.
Create in a pending state. Write the workspace with status = 'provisioning'. Other services ignore pending resources. On failure, mark it abandoned; a cleanup job deletes it later. The compensating action becomes a state change you fully control, not an API call to a vendor.
Soft-delete first, hard-delete later. An agent that archives a document can be compensated by un-archiving. An agent that permanently deletes it cannot. Make destructive operations two-phase.
Defer irreversible steps. Emails, webhooks, SMS, and shipments should happen after the pivot, triggered from an outbox once the run is committed. This is the single highest-leverage change for agent safety: it converts "we sent an email we can't unsend" into "we never enqueued the email."
Pass an idempotency key on every write. The Idempotency-Key header lets the downstream service recognize a retry and return the original result instead of performing the action twice. The IETF draft that specified it has expired and it is not a formal standard, but the pattern is widely implemented and worth requiring in your tool contracts. Derive the key from run_id + step_no + action, not from a random value per attempt, or retries will not deduplicate.
def idempotency_key(run_id: str, step_no: int, action: str) -> str:
return f"{run_id}:{step_no}:{action}"
A reservation, concretely
The reservation pattern is worth spelling out because it changes the failure story more than any prompt ever will. Instead of calling purchase_seats(workspace, 40), the agent calls reserve_seats(workspace, 40, ttl=900). The billing service writes a row with an expiry and returns a reservation id.
create table seat_reservations (
id uuid primary key,
workspace_id uuid not null,
quantity int not null check (quantity > 0),
state text not null default 'held', -- held | confirmed | released | expired
expires_at timestamptz not null,
created_at timestamptz not null default now()
);
-- Availability excludes holds that have not expired yet.
select
(select seats_purchased from workspaces where id = $1)
- coalesce((
select sum(quantity) from seat_reservations
where workspace_id = $1
and state = 'held'
and expires_at > now()
), 0) as seats_available;
The agent's workflow now has a step that cannot leave a lasting mess. If the run fails after reserving, the compensation is release_seats(reservation_id), and if that call is lost, the reservation expires on its own. The worst case degrades from "customer is billed for seats they never used" to "seats are unavailable for fifteen minutes."
Two details make this work in practice. The availability query must ignore expired holds, or a crashed run will block inventory until someone notices. And confirmation must be a conditional update from held to confirmed, returning zero rows if the hold already expired — otherwise a slow run confirms a reservation that has already been released, and you have sold the same seat twice.
The same shape applies beyond seats: a file upload can reserve a storage quota, an outbound shipment can reserve inventory, an API key can be issued in a pending state. The pattern is not specific to billing. It is how you make a step fail-safe when you cannot make it transactional.
The ledger is the source of truth
An agent cannot reliably remember what it did. Context gets truncated, the process restarts, and the model's own account of its actions is a narrative, not a record. Compensation must be driven by a durable log written by the code that performed the action, not by the model's memory.
create table action_log (
id bigserial primary key,
run_id uuid not null,
step_no int not null,
action text not null,
target text not null,
request jsonb not null,
response jsonb,
state text not null, -- attempted | committed | failed | compensated | compensation_failed
reversible boolean not null default true,
created_at timestamptz not null default now(),
updated_at timestamptz not null default now(),
unique (run_id, step_no, action)
);
create index action_log_run_idx on action_log (run_id, step_no);
Write the attempted row before the call, not after. If the process dies mid-call, the row is the only evidence that something may have happened, and it is exactly the row you need for reconciliation. An attempted row older than the timeout window is a shape-3 unknown, and a reconciler should resolve it.
The unique (run_id, step_no, action) constraint is doing real work. It prevents the same step from being logged twice when a worker retries after a crash, which in turn prevents the compensator from trying to undo the same effect twice.
from dataclasses import dataclass
from typing import Awaitable, Callable
@dataclass
class Step:
no: int
name: str
forward: Callable[["Ctx"], Awaitable[dict]]
compensate: Callable[["Ctx", dict], Awaitable[None]] | None = None
async def run_saga(run_id, ctx, steps: list[Step]):
completed: list[tuple[Step, dict]] = []
for step in steps:
await log_attempt(run_id, step)
try:
result = await step.forward(ctx)
except Exception as exc:
await log_failed(run_id, step, exc)
await compensate(run_id, ctx, completed)
raise
await log_committed(run_id, step, result)
completed.append((step, result))
The compensator walks the completed list in reverse and records every outcome:
async def compensate(run_id, ctx, completed):
for step, result in reversed(completed):
if step.compensate is None:
await record_manual_intervention(run_id, step)
continue
try:
await step.compensate(ctx, result)
except Exception as exc:
await log_compensation_failed(run_id, step, exc)
await enqueue_compensation_retry(run_id, step, attempt=1)
else:
await log_compensated(run_id, step)
Two design choices in that snippet are deliberate. A step without a compensator does not silently pass; it records a manual intervention, which surfaces in an operator queue. And a compensation failure does not abort the loop. Remaining compensations still run, because leaving several steps half-undone is worse than leaving one.
Ordering is a dependency problem, not just "reverse"
The textbook rule is to compensate in reverse order. That is correct for a linear chain where each step depends on the previous one, and wrong when it does not.
Consider: the agent creates a project (step 1), attaches a repository (step 2), and invites a team (step 3). If step 4 fails, undoing step 1 first will fail because the project still has a repository attached. You must detach step 2 before deleting step 1. Reverse order happens to work here, but only because the dependencies are aligned with the sequence.
Now consider: the agent uploads a file to object storage (step 1) and creates a database row pointing at it (step 2). Reverse order deletes the row, then the file. Fine. But if the file upload is step 2 and the row is step 1, reverse order deletes the file first, leaving a row pointing at nothing for a few seconds — which a background job might read. The correct order depends on which resource is referenced, not on which came first.
The practical approach is to declare dependencies explicitly:
@dataclass
class Step:
no: int
name: str
forward: Callable
compensate: Callable | None = None
depends_on: tuple[int, ...] = ()
Then compensate in reverse topological order. In most agent workflows the graph is nearly linear, so the extra machinery is small. Where it matters, it matters a lot.
Money is the case that punishes optimism
Payment flows concentrate every hard problem: irreversibility, external state, regulatory constraints, and customer-visible effects.
The most important design decision is to separate authorization from capture. Authorizing a card places a hold; capturing moves the money. If the agent authorizes and a later step fails, voiding the authorization is a clean compensation that never touches the customer's balance in a lasting way. If the agent captures immediately, the only compensation is a refund, which takes days to appear and can fail for reasons outside your control.
authorize ──> (reversible: void)
capture ──> (reversible only via refund, slow, may fail)
Three rules follow.
Never compensate a capture with a negative charge. A "negative charge" is not a refund. It creates a second transaction with its own fees, its own dispute risk, and a confusing statement descriptor. Use the payment provider's refund API, and model refunds as their own events in your ledger.
Track partial refunds explicitly. If the agent charged $500 and only $200 of work succeeded, the compensation is a $300 refund, not a full reversal. Your ledger needs the original amount, the compensated amount, and the reason, or reconciliation will never balance.
Reconcile against the provider, not your own records. Webhooks arrive late, out of order, and sometimes not at all. A nightly job that lists recent charges and refunds and compares them to the action log catches the cases where your compensation call succeeded at the provider but failed before you recorded it.
💡 Practical note: "compensated" in your database should mean "the downstream provider confirmed the reversal," not "we sent the request." Those are different states and should have different names.
Irreversible steps need a holding pattern
Email is the classic example. Once sent, it exists in a mailbox you do not control. You can send a correction, but you cannot retract the first message. If an agent sends a "your account is ready" email and the account creation then fails, you have lied to a customer.
The fix is architectural: send nothing until the run is committed. Write the message to an outbox in the same transaction as the final state change, and let a separate process deliver it after the pivot. If the run fails, the outbox row is deleted or marked cancelled and no message ever exists.
Webhooks to customer endpoints deserve the same treatment. A webhook is a promise that an event happened. Emitting project.created and then deleting the project produces a customer-side inconsistency you cannot fix from your side.
For actions that genuinely cannot wait — a physical shipment, a scheduled call, an SMS verification code — require an explicit confirmation step and record who approved it. That approval is not a formality; it is the compensating action's substitute, and it belongs in the audit log.
The reconciler is not optional
Every design so far assumes you can find out what happened. In practice, some handoffs lose the answer: a process is killed between the provider's commit and your log write, a webhook never arrives, or a timeout leaves an attempted row with no outcome.
A reconciler is a scheduled job that resolves those rows by asking the downstream system. It is not glamorous, and it is the piece teams skip until the first customer complaint.
The reconciler works because every write carried a stable identifier. For each attempted row older than the timeout window, it queries the provider by idempotency key or natural key and asks one question: did this effect land?
for each action_log row where state = 'attempted' and age > timeout:
outcome = provider.lookup(idempotency_key = run_id || step_no || action)
if outcome == found:
mark committed; continue forward or compensate per pivot
if outcome == not_found:
mark failed; safe to retry
if outcome == still_pending:
leave alone; alert if age > hard_limit
Three operational details matter. Run the reconciler on a schedule measured in minutes, not days, because the window between "unknown" and "customer notices" is short. Give it a read-only credential so a bug in reconciliation cannot perform writes. And alert on the rate of unresolved rows, not only on individual ones — a sudden rise usually means a provider changed its error behavior or your timeout is shorter than its p99.
The reconciler also gives you something valuable for free: a daily count of effects that landed without a matching local record. That number is the honest measure of how much you are guessing, and it should trend toward zero.
Keep the agent out of the compensation decision
This is the part that separates agent workflows from ordinary sagas. In a normal saga, the orchestration code decides what to compensate. When an LLM is in the loop, there is a temptation to let it decide: "The seat purchase failed; please clean up anything you created."
That is the wrong shape for three reasons.
First, the model does not have a reliable view of what committed. Its context may have been truncated, and it may believe a failed call succeeded or vice versa. Second, compensation is a policy decision with business consequences — refunds, deletions, customer communication — and policy belongs in code that can be reviewed and tested. Third, an agent asked to "clean up" may take additional actions you did not enumerate, which is exactly the excessive-agency problem you were trying to contain.
The right division of labor:
- The model proposes a plan using a constrained tool schema.
- The orchestration layer executes each step and writes the ledger.
- The orchestration layer owns compensation, driven by the ledger and a declared compensator per step.
- A human resolves anything the compensator cannot.
If the model needs to react to a failure, give it a narrow tool: request_human_review(reason), not execute_arbitrary_cleanup().
Test the half, not just the whole
Most test suites exercise the happy path and one total failure. The half-success cases live in between, and they are where the bugs are.
Build a fault matrix. For an N-step workflow, run the following:
| Scenario | Expected behavior |
|---|---|
| Fail at step k, for every k | Steps 1..k-1 compensated in reverse dependency order |
| Fail compensation at step j | Remaining compensations still run; j is retried and escalated |
| Timeout at step k (unknown outcome) | Reconcile before retry; no duplicate effect |
| Duplicate delivery of step k | Idempotency key returns the original result; one effect |
| Process killed after commit, before log | Ledger row recovered from attempted; reconciler resolves |
| User cancels mid-run | Compensation runs; no further forward steps |
That is six scenarios times the number of steps, which sounds like a lot until you compare it to a week of manual cleanup after one bad onboarding.
Fault injection is easiest when the fake environment can be told to fail at a specific step. If your agent can only be tested end-to-end against real services, add a fail_at hook to the tool wrapper in staging — one line that raises after recording the call. The test then asserts on the ledger, not on the agent's message.
The decision framework
Before giving an agent a write tool, answer these five questions in writing.
- What is the pivot? Name the step after which the workflow is considered committed. Everything before must be reversible; everything after must be forward-only.
- What compensates each step? If the answer is "nothing," either the step moves after the pivot or it requires human approval.
- Is each compensation idempotent? It will be retried. Derive its key from the run and step, not from the attempt.
- How will you detect a shape-3 unknown? You need a record written before the call and a reconciler that can query the downstream state.
- Who gets paged when a compensation fails? If the answer is "nobody," the design is incomplete.
A useful sanity check: walk the ledger for a failed run and ask whether a support engineer could reconstruct exactly what happened, in what order, and what remains undone. If they need to read the model's chat transcript to answer that, the ledger is not doing its job.
Compensating actions are not a safety net around the agent. They are the mechanism that makes partial failure a normal, recoverable state instead of an incident. The teams that get this right tend to stop thinking about "agent reliability" as a model problem and start treating it as workflow engineering — which is what it is.