How to recover an AI workflow that failed after charging a customer

The customer was charged. The AI workflow crashed before the order was marked paid, the confirmation email was sent, or the support ticket was updated. The dashboard says “failed.” The payment provider says “succeeded.” Retrying the whole agent run may charge the customer again.

This is not an AI hallucination problem. It is a partial-commit problem across systems: the payment provider committed one side effect, while your application did not finish the rest of the workflow. The model may have contributed to the plan, but recovery must come from durable operation state, idempotency, reconciliation, and a clear rule for what is safe to repeat.

The first priority is not to make the agent continue. It is to establish what already happened. This article gives a recovery sequence for a workflow that failed after charging, then shows how to design the state machine and tests so the next failure does not become a duplicate payment or a confusing customer experience.

Stop before retrying the whole workflow

When a workflow reports failure after a payment attempt, freeze automatic retries for the payment operation until its state is known. A retrying worker may see “order not paid” in your local database and create a second charge, even though the provider accepted the first one.

Distinguish three facts:

What the application intended: the user approved a purchase or refund.
What the provider did: a payment operation may have been created, authorized, captured, declined, or left pending.
What the rest of the workflow completed: order status, inventory, confirmation, ledger entry, and support notification.

The local order row is not authoritative for the remote charge if the worker crashed before updating it. The error message is not proof that the payment failed. The next action should be reconciliation: query the payment provider using the stored provider reference or the original idempotency key, inspect verified webhook events, and compare the resulting state with your local operation record.

Do not ask the model to infer whether the charge went through from a timeout or a log excerpt. That is a deterministic reconciliation task against the payment system of record.

Model the purchase as several operations

“Checkout succeeded” often hides multiple side effects:

authorize or capture payment
reserve inventory
create the order
record the ledger entry
send confirmation
update the support or CRM record

These steps do not share a single database transaction. The payment provider, inventory service, and email system each commit independently. If step four fails after step one, the workflow is partially complete, not simply failed.

Give each operation a durable identity and state. A simplified purchase record might include:

purchase_id
payment_operation_id
provider_payment_reference
payment_state
order_state
inventory_state
notification_state
last_reconciled_at
recovery_owner

The state machine should represent uncertainty explicitly:

payment_pending → payment_succeeded → order_pending → order_created
       ↓                 ↓                 ↓
   payment_failed   reconciliation    recovery_required

If the payment state is unknown, do not move the workflow back to payment_pending and create a new charge. Move it to reconciliation_required, suppress duplicate payment attempts, and let a recovery worker or operator inspect the provider’s state.

Reconcile with the provider’s source of truth

For an unknown payment outcome:

Retrieve the provider-side payment using the reference stored before the call, if available.
Check signed webhook or event records and deduplicate them by provider event ID.
Confirm the amount, currency, customer, and merchant reference match the intended purchase.
Determine whether the payment is authorized, captured, declined, cancelled, or still pending.
Persist the provider state and reconciliation evidence locally.
Continue only the steps that remain incomplete.

Do not use an unsigned callback or a user-provided screenshot as the sole evidence of payment. Verify webhook signatures and match the event to the expected account, amount, currency, and operation. A valid payment for a different order is not a successful result for this order.

Provider APIs have different state machines and retention rules. Follow the exact provider contract for the endpoint you use. Stripe’s idempotency documentation, for example, explains that API v1 requests with the same key and parameters can replay the first result, and that keys may be removed after they are at least 24 hours old. After the provider’s retention window, reusing the key may create a new request; reconcile the business object before retrying. (Stripe, idempotent requests)

Use stable idempotency keys and durable operation records

Create an idempotency key once per logical payment operation and persist it before sending the first provider request. Reuse it for retries of that same operation with the same parameters. A key generated inside the retry loop defeats the purpose.

The key should be opaque and scoped. Do not use a customer email, card number, or raw prompt. A UUID tied to a stored payment_operation_id is usually easier to manage than reconstructing intent from a string.

Before each retry, compare a canonical hash of the request parameters with the stored operation. If the amount, currency, recipient, or order changed, stop and require a new operation or explicit approval. The same key must not silently refer to a different charge.

Your operation table needs statuses such as pending, in_progress, succeeded, failed, and unknown. unknown is not an error to be retried blindly; it is a state that triggers reconciliation. Store the provider request ID, safe response fields, idempotency key, amount, currency, and relevant timestamps. Never store payment card data in general logs or AI traces.

Resume from the last completed step

Once the payment is confirmed, resume the workflow from the next incomplete operation. Do not restart the agent from the original prompt and let it decide what to do again. A fresh model run may choose a different payment amount, create another order, or send a duplicate message.

Use a deterministic orchestrator for payment recovery:

def recover_purchase(purchase_id, payments, orders, inventory, notifications):
    purchase = load_purchase(purchase_id)
    payment = payments.reconcile(purchase.payment_operation_id)

    if payment.state == "unknown":
        mark_recovery_required(purchase_id, reason="payment_state_unknown")
        return "needs_reconciliation"

    if payment.state == "failed":
        mark_payment_failed(purchase_id, payment.reason_code)
        return "payment_failed"

    if payment.state != "succeeded":
        return "payment_pending"

    if not purchase.order_created:
        orders.create_once(
            purchase_id=purchase.id,
            payment_reference=payment.reference,
            idempotency_key=f"order:{purchase.id}",
        )

    if not purchase.inventory_committed:
        inventory.commit_once(
            purchase_id=purchase.id,
            idempotency_key=f"inventory:{purchase.id}",
        )

    if not purchase.confirmation_sent:
        notifications.send_once(
            purchase_id=purchase.id,
            idempotency_key=f"confirmation:{purchase.id}",
        )

    mark_purchase_complete(purchase_id)
    return "complete"

This is illustrative pseudocode. The create_once and send_once names represent application-owned wrappers that must persist operation state and use downstream idempotency where supported; they are not generic provider methods. In a real system, each step should return a durable result and distinguish failed from unknown outcomes.

The model can still help explain the failure, draft a customer message, or classify a support ticket. It should not decide whether to charge again or whether a payment is complete.

Reconcile webhooks and request responses as one event stream

Payment providers may send webhooks before or after the synchronous API response, and they may redeliver the same event. Build one event ingestion path that verifies signatures, persists provider event IDs, and applies state transitions idempotently. Do not let the webhook handler and the request handler independently create orders or send notifications.

The event processor should compare the provider event with the stored payment operation: account, payment reference, amount, currency, and intended purchase. If the event is valid but does not match the operation, quarantine it for investigation. A signed event proves that the provider sent it; it does not prove that it belongs to this customer’s workflow.

Use provider timestamps and event IDs for deduplication, but query the canonical payment object when events arrive out of order or conflict. A late pending event should not overwrite a later succeeded state. The exact transition rules depend on the provider’s state machine; encode them as deterministic code and test every permitted and forbidden transition.

If the provider’s API is unavailable during reconciliation, keep the operation in unknown and retry the read or wait for a verified webhook. Do not retry the charge merely because status lookup failed. User-facing copy should distinguish “we cannot confirm yet” from “payment failed.”

Decide whether to compensate or complete the workflow

Once the provider confirms a charge, there are two broad recovery paths:

Complete forward: finish order creation, inventory commitment, and notifications. This is often right if the purchase remains valid and the missing work is operational.

Compensate: refund or void the payment and cancel the order if fulfillment cannot proceed or the transaction violates policy.

Compensation is a new financial operation, not deletion of the original. It needs its own idempotency key, authorization, audit record, and reconciliation. A refund may itself time out or remain pending. Do not mark the customer “refunded” until the payment provider confirms the refund state.

Some actions cannot be compensated completely. An email cannot be unsent; a warehouse may already have shipped the item. Design the saga with business owners: define which steps can be retried, which require compensation, and which need human review. The agent should not invent a compensation plan after a partial failure.

Handle the order and payment state transition atomically where possible

Within your own database, record the local order transition and an outbox event in one transaction. A worker can then process the remaining side effects. The outbox provides reliable delivery from the database to a worker, but it does not replace payment-provider idempotency.

For example, after reconciling a confirmed charge, one transaction can:

mark the payment operation as succeeded;
mark the purchase as ready for fulfillment;
insert an outbox event for order completion;
commit.

If the worker crashes, the event is redelivered. The order and inventory handlers use stable operation keys so redelivery does not duplicate the effect. If a second payment attempt is accidentally enqueued, the payment operation record prevents it from being treated as a fresh purchase.

Communicate honestly with the customer

The customer should not have to infer the payment state from an error screen. Use specific, truthful states:

“Payment is being confirmed. Don’t submit again; we’ll update this order shortly.”
“Your payment succeeded, but order confirmation is delayed. Your order reference is …”
“We could not confirm the payment yet. We’re checking with the payment provider before retrying.”
“The payment failed and no order was created. You can review the payment method and try again.”
“The payment was reversed; the refund is processing. The provider’s status will be updated here.”

Use the exact status your application can verify. If it is unknown, say so without implying success or failure. Preserve the purchase reference and provide a support route. Do not claim a refund will arrive instantly unless the provider guarantees that timing.

If you need human review, show that the case is being handled and provide an expected response window only if the team can meet it. Keep customer-facing copy consistent with the durable operation state so the UI and support dashboard do not contradict each other.

Avoid logging sensitive payment details

Recovery needs enough data to identify the operation, but not raw card data. Store provider references, opaque operation IDs, amounts, currency, safe status codes, and timestamps. Keep the payment service responsible for payment credentials and avoid sending card numbers or security codes through model prompts, tools, or traces.

Restrict access to recovery dashboards. Support may need to see the payment state and safe identifiers but not all provider payloads. Audit manual refunds and overrides. Redact secrets before exporting traces, and set retention according to the sensitivity and operational need.

An AI trace is not a payment ledger. The financial record should live in the payment and accounting systems, with the workflow linking to those records by stable IDs.

Keep the incident process separate from customer support improvisation

When a customer reports a charge without an order, support should have a runbook that starts with the operation and provider references, not with a button that lets the agent “try checkout again.” The runbook should identify who can approve a refund, how to verify the provider state, and how to handle an amount or currency mismatch.

If the issue is widespread, pause new payment attempts for the affected workflow while leaving safe read-only status checks available. Do not disable all of checkout if only the post-payment order writer is broken; instead, make the system’s uncertainty visible and prevent new charges until it can create or reconcile orders reliably.

For each recovery decision, record whether the team completed forward, compensated, or required manual reconciliation. Add the final outcome to an incident review and a fault-injection test. Without that step, the same failure boundary can recur after the next queue or provider change.

Distinguish authorization, capture, void, and refund

Payment systems may separate an authorization from a capture. An authorization can reserve funds without completing settlement; a capture requests the charge; a void releases an uncaptured authorization; a refund reverses a captured payment. The exact lifecycle and names differ by provider and payment method, so use the provider’s documented state machine rather than treating every success response as “money settled.”

Your workflow should record which stage it reached. If fulfillment fails while the payment is only authorized, a void may be appropriate. If the charge was captured, the recovery may require a refund. Repeating a capture after a lost response can charge twice if the request is not idempotent. These are business and accounting decisions, not choices for the model to make from a vague error message.

Coordinate the recovery policy with finance and operations. Some payment methods settle asynchronously, and a refund can take time to appear for the customer. The application should display the verified provider state and the expected next step without promising a settlement date that the provider does not guarantee.

If your provider exposes a reconciliation report or settlement export, compare it with the local operation ledger on a schedule. This catches cases where both the UI and worker lost the same acknowledgment but the provider ultimately completed the charge.

Prevent the interface from creating a second purchase

While a payment outcome is pending or unknown, disable the control that would start a new purchase, but keep a status view available. The user should see the original operation reference and be able to refresh its state. If they change the cart or payment method, create a separate purchase intent and make clear that the earlier operation is unresolved; do not silently re-use or replace its idempotency key.

After reconciliation, unlock the appropriate action. If the payment succeeded and the order exists, show the order. If it succeeded but the order was not created, continue recovery or provide a support path. If it definitively failed, let the customer submit a new payment attempt. If the state remains unknown, keep the warning visible and avoid charging again until the provider or an operator resolves it.

This behavior should work across refreshes, tabs, and devices. A disabled button in one browser is not enough if the same customer can submit the purchase from another session. Enforce the pending-operation check server-side using the stable purchase ID. Do not rely on client state as the financial control.

Test the failure window deliberately

The dangerous failure often occurs between remote success and local acknowledgment. Test that exact window:

Provider accepts the payment, then the client connection drops.
Provider returns success, then the worker crashes before updating the database.
Webhook arrives before the synchronous response.
Queue redelivers after local state says in_progress.
Provider lookup is temporarily unavailable during reconciliation.
Order creation succeeds but inventory reservation fails.
Refund succeeds but the confirmation email fails.

Assert the number of provider-side charges, orders, inventory commitments, and notifications—not merely the final API response. Verify that unknown outcomes route to reconciliation and that a second worker cannot create a fresh payment key for the same purchase.

Run these tests with the same orchestration and idempotency code used in production. A unit test that mocks the payment provider as a simple success function will not expose the failure window.

Monitor recovery as its own workflow

Track:

payments in unknown state and their age;
time from provider success to local order completion;
duplicate attempt and idempotency replay counts;
orders requiring compensation;
refund and void outcomes;
manual intervention volume;
notification delays after confirmed charge;
customer support contacts related to ambiguous payment state.

Alert on operations stuck beyond the expected recovery window. A payment that is confirmed but has no order is different from a payment whose outcome is unknown; create separate queues and runbooks for them.

When an incident occurs, preserve the operation ID, provider reference, task trace, and state transitions. Identify whether the fault was in model planning, orchestration, provider communication, local persistence, or recovery policy. Add a regression test for the failure boundary and update the workflow state machine—not just the prompt.

A recovery checklist

When an AI workflow fails after charging a customer:

Pause automatic retries for the payment operation.
Resolve the stable operation ID and original idempotency key.
Query the payment provider or reconcile a verified webhook.
Confirm amount, currency, customer, and purchase reference.
Record the provider’s actual state locally.
Resume only the incomplete downstream steps, each with its own idempotency.
Compensate only when policy requires it, using a separate refund operation.
Communicate the verified state to the customer.
Preserve a safe audit trail and alert on unresolved outcomes.
Add the failure window to automated tests.

The key rule is simple: a failed workflow is not proof that the payment failed. Once money may have moved, recovery starts by reconciling the provider’s state and resuming from durable operation records. Let the model help with language and analysis; let deterministic workflow code control what gets charged, what gets reversed, and what the customer is told.