The hidden cost of retries in AI API calls

A retry looks like a second chance. On a metered model API, it is usually a second invoice.

The sequence is familiar. A request takes longer than expected. The client times out at 30 seconds and retries. The second attempt succeeds in 12 seconds. The user gets one answer. The provider may have generated two, and your application may have paid for both. If the first request was still running when the client gave up, the provider could have finished it after you stopped listening.

Retry logic is necessary. Networks fail, rate limits happen, and providers have bad minutes. But the standard advice — exponential backoff, jitter, a maximum attempt count — was written for APIs where a failed request costs approximately nothing. A generative endpoint charges for prefill, often for output, sometimes for reasoning tokens, and sometimes for work that completes after the client has abandoned it. The retry policy has to account for that.

This article is about the cost side of retries: which failures are free, which are billable, why timeouts are the most expensive, how layered retries multiply, and how to build a retry path that is bounded by money and time rather than hope.

A retry is a new request

Unless the provider implements idempotency for the endpoint, a retry is a fresh inference request. It resends the entire prompt: system instructions, tools, retrieved documents, conversation history, and the user's message. The provider processes it again. If the first attempt already produced tokens, those tokens are gone from your client but not necessarily from the bill.

This is the first mental correction. People compare an LLM retry to retrying a database query. A failed SELECT is cheap to repeat. A failed 20,000-token generation is not. The prompt is the payload, and the payload is billed on every attempt.

The second correction follows: a successful retry does not erase the cost of the failed attempt. If a task succeeds on the third try, the cost of the task is the sum of three attempts, not the cost of the successful one. Your unit economics should use cost per successful task, not cost per successful request. The difference is the retry tax.

A simple formula:

attempts_per_success = total_attempts / successful_tasks
effective_cost_per_success = average_attempt_cost × attempts_per_success

If a feature normally costs $0.02 per attempt and averages 1.4 attempts per success, the effective cost is $0.028 per successful task — a 40 percent increase that never appears on a per-request dashboard. If a bad provider day pushes attempts per success to 3, the effective cost triples.

Track that ratio per endpoint, per model, and per prompt version. A rising ratio is an early warning that something upstream is degrading, often before the error rate looks alarming.

Free failures, ambiguous failures, billable failures

Not all failures cost the same. A useful classification:

Failure	Was work done?	Cost	Retry?
DNS failure, connection refused, TLS error	No request reached the provider	Effectively free	Yes, promptly
429 before admission	Rejected at the door	Free	Yes, honor `Retry-After`
400, 401, 403, 404, 422	Request invalid or unauthorized	Free	No, fix the request
Fast 5xx before generation	Likely no meaningful work	Probably free	Yes, with backoff
Slow 5xx after tens of seconds	Generation may have completed	Likely billable	At most once, and count it
Read timeout while streaming	Generation may still be running	Likely billable for the full output	Cancel explicitly; consider fallback
Connection reset mid-stream	Partial output produced	Billable for what was generated	Maybe, if the partial is unusable

The word "likely" is doing real work. You often cannot know whether a specific failed attempt was billed until usage data or an invoice arrives. The retry policy should assume the pessimistic case and reconcile later, not assume the best case and discover the cost at the end of the month.

This table also explains why a single global retry policy is a bad idea. A 400 from a malformed tool schema and a 429 from a rate limit are both "errors," but one should never be retried and the other should be retried after a specific delay. A 500 returned in 200 milliseconds is a different event from a 500 returned after 45 seconds of generation.

Google's Vertex AI retry guidance draws a similar line: retry transient conditions such as 408, 429, and 5xx, do not retry permanent client errors, use exponential backoff with jitter, cap attempts, and consider idempotency before retrying (Google Cloud, retry strategy). The nuance for generative endpoints is that "safe to retry" is not the same as "free to retry."

The timeout is the expensive part

A client-side timeout stops you from waiting. It does not necessarily stop the provider from generating.

If the client abandons the socket without an explicit cancellation, the server may continue to completion. You never see the output, but you may be billed for it. Retry twice on a slow provider and you can pay three times for one answer you never received.

Three mitigations matter, in order:

Cancel explicitly. Use the SDK's cancellation or abort mechanism when a timeout fires. Closing the connection is not always the same as canceling the request. If the provider supports a cancellation endpoint or a request-level abort, use it.

Set timeouts from a real distribution. A round-number timeout below your own p99 turns normal slow requests into billable failures. Measure time-to-first-token and total latency for the workload, then set the timeout above the tail you are willing to tolerate. A timeout that is too aggressive is a cost multiplier disguised as a reliability setting.

Prefer a fallback to a repeated retry after a timeout. If a route was slow enough to time out once, the same route is not the most likely to be fast on the second attempt. A different model, a shorter prompt, or a queued path may cost the same as another attempt and have a better chance of succeeding.

There is a subtle interaction with streaming. A stream that has already delivered part of an answer is not free to abandon; you may have paid for the tokens produced so far. If the partial answer is usable, consider returning it with a clear "incomplete" state instead of paying for a full regeneration. If it is not usable, cancel and count the partial attempt against the task budget.

🚨 Production warning: A timeout is not proof that the request failed. It is proof that you stopped waiting. Treat the outcome as unknown until you reconcile it with usage data or the provider's request logs. Retrying an unknown outcome is how duplicate side effects and double billing happen.

Layered retries multiply quietly

Most production systems have more than one retry layer:

the provider SDK retries transient errors;
the application retries the SDK call;
the job queue redelivers the task;
a proxy or gateway retries upstream;
a workflow engine retries the activity.

Each layer is reasonable in isolation. Together they multiply. If the SDK allows 3 attempts, the application allows 3, and the queue redelivers 3 times, one user action can produce 27 requests. The provider sees 27 attempts, and your budget sees 27 chances to pay.

The fix is ownership. Exactly one layer should own the retry budget for a given operation, and the other layers should be configured not to add their own. Temporal's AI guidance makes this explicit: when a workflow engine is retrying activities, disable the provider client's transport retries so you don't get more attempts than intended, and handle Retry-After rather than ignoring it with generic exponential backoff (Temporal, AI reference architecture).

A practical ownership model:

Interactive request path: the application owns retries, with a deadline tied to the user's patience. The SDK is configured with zero or one retry.
Queued job path: the queue or workflow engine owns retries, with a persistent attempt counter. The worker does not retry internally.
Batch path: the batch orchestrator owns retries, with a per-item budget and a cap on total spend.

Write the ownership down. "The SDK does it" is not a policy if three other layers also think they own it.

Backoff, jitter, and the retry storm

Exponential backoff with jitter is still the right default for transient failures. The details matter more than the slogan.

Exponential growth prevents hammering a service that is already struggling.
Full jitter (a random delay between zero and the computed backoff) spreads retries across clients.
A maximum delay prevents absurd waits.
A maximum attempt count bounds the work.
A deadline prevents sleeping past the point where the result is useful.
Retry-After takes precedence when the server provides it.

The last two are the ones teams skip. Backing off for eight seconds into a deadline that expires in two is pure waste. Honoring a Retry-After of 30 seconds when the user's request has a 10-second budget is also waste, unless the request can be handed to a queue.

A retry storm is what happens when many clients retry at once. The provider rate-limits, clients back off, the backoff expires, and the retries arrive together again. Jitter helps, but it does not fix a system that is retrying work it should have shed. If the error rate is high and the queue is growing, the correct action is load shedding, not more retries. Return a degraded response, queue the work, or fail fast with a clear message.

Retrying the model call is not the same as retrying the side effect

A model call can be retried more safely than the action it proposes. The dangerous case is retrying the whole operation, including the tool call, the database write, the email, or the payment.

Consider an agent that books a meeting. The model call succeeds, the calendar API call succeeds, and then the response is lost on the way back. The client retries the whole operation. The model may choose to book again. Now there are two meetings.

The general pattern is to separate the probabilistic proposal from the deterministic side effect:

The model proposes an action as structured data.
Code validates the proposal against schema and policy.
Code executes the action with an idempotency key derived from the logical operation.
The result is stored and returned for any retry with the same key.

This is standard distributed-systems advice, but AI agents make it urgent because the proposal is generated fresh on every attempt. A retry doesn't replay the same decision; it may produce a different one. An idempotency key on the model call does not protect the side effect unless the side effect uses its own key.

If the downstream API supports idempotency, use it. Stripe's API, for example, retains idempotency keys for at least 24 hours, replays the stored response for a repeated key with identical parameters, and treats a connection error as an indeterminate outcome that should be reconciled rather than blindly repeated (Stripe, idempotent requests). The retention window matters: a retry after the key is pruned can execute a new operation.

Do not assume every LLM provider or tool API supports idempotency keys. Many don't. If the downstream action is not idempotent, the application needs a state machine: pending, in_progress, succeeded, failed, unknown. A retry can resume a known state; it should not blindly repeat an unknown one.

Idempotency keys need a scope and a retention policy

An idempotency key is only useful if it identifies the same logical operation across retries and process restarts.

Bad keys:

a random UUID generated inside the retry loop;
a timestamp;
the user's email address;
a hash that includes the attempt number;
a key generated after the first request has already started.

A good key is stable for the operation and scoped to the tenant or account. For a refund, refund:{account_id}:{order_id}:{refund_request_id} is better than a fresh UUID. For a generation, generation:{conversation_id}:{client_message_id} lets a retried request return the original answer instead of generating a second one.

Retention is part of the contract. If the key expires after an hour and the retry happens after two, the operation runs again. Stripe's 24-hour window is a useful reference point, not a universal rule. Choose a window longer than your worst-case retry and reconciliation path, and document it.

Streaming complicates replay. Some idempotency implementations refuse to store streamed responses because an SSE body cannot be meaningfully replayed from storage. If your feature streams, decide what a retry means: resume from the last token, return the stored final text, or start a new generation with a new key. Each choice has a different cost and consistency profile.

Budget the operation, not just the attempt count

A maximum attempt count treats a 200-token classification and a 50,000-token document analysis the same. That is a poor proxy for cost.

Add a spend bound and a deadline to the retry policy:

from dataclasses import dataclass
from time import monotonic, sleep
import random

@dataclass
class RetryPolicy:
    max_attempts: int = 3
    max_spend_usd: float = 0.10
    deadline_seconds: float = 45.0
    base_delay: float = 0.5
    max_delay: float = 8.0

class RetryableError(Exception):
    def __init__(
        self,
        message: str,
        retry_after: float | None = None,
        estimated_cost_usd: float = 0.0,
    ):
        super().__init__(message)
        self.retry_after = retry_after
        self.estimated_cost_usd = estimated_cost_usd

def full_jitter(attempt: int, policy: RetryPolicy) -> float:
    ceiling = min(policy.max_delay, policy.base_delay * (2 ** (attempt - 1)))
    return random.uniform(0, ceiling)

def with_retries(call, policy: RetryPolicy):
    spent = 0.0
    started = monotonic()
    attempt = 0

    while True:
        attempt += 1
        remaining = policy.deadline_seconds - (monotonic() - started)
        if remaining <= 0:
            raise TimeoutError("deadline exceeded before attempt")

        try:
            result, _ = call(attempt)
            return result
        except RetryableError as error:
            spent += error.estimated_cost_usd

            if attempt >= policy.max_attempts:
                raise
            if spent >= policy.max_spend_usd:
                raise RuntimeError("retry budget exhausted") from error

            delay = error.retry_after or full_jitter(attempt, policy)
            if delay >= policy.deadline_seconds - (monotonic() - started):
                raise TimeoutError("next retry would exceed deadline") from error

            sleep(delay)

The example assumes call returns a value and a cost, and that the caller classifies errors into RetryableError versus a non-retryable exception. That classification is the important part; the loop is ordinary.

Three guards are easy to miss:

Spend is incremented for billable failures, not only successful attempts.
The deadline is checked before sleeping, so the retry doesn't wake up after the result is useless.
retry_after takes precedence over computed backoff.

A token budget is a useful companion. If the request is large and the provider bills cached input differently, estimate the attempt cost from the prompt size before sending, and refuse an attempt that would exceed the remaining budget. You may not know the exact output length in advance, but you can bound the input cost.

Reconcile unknown outcomes

The hardest retry state is "unknown." The request may have succeeded, failed, or still be running. The client has no response. Retrying can duplicate work; not retrying can leave a user waiting for something that already happened.

A reconciliation path needs:

a stable operation ID that is recorded before the request is sent;
the provider's request ID, if it is returned in headers or logs;
a usage record that can be matched to the operation;
a way to query the downstream state (did the refund post? did the email send?);
a policy for how long to wait before declaring the outcome unrecoverable.

For model calls, the provider's usage dashboard and request logs may be the only way to know whether a timed-out attempt was billed. For tool calls, the downstream system is the source of truth. For a database write, a unique constraint plus a read-back after a conflict is usually enough.

Record the outcome as one of succeeded, failed, cancelled, or unknown. Do not collapse unknown into failed. That is how a retry becomes a duplicate charge.

Retry budgets for queues

Queued work changes the calculus. A user isn't waiting on the connection, so a longer backoff is acceptable. But the queue can also hide a retry storm: the number of jobs grows, workers pick up the same job again, and the bill rises without a corresponding rise in completed tasks.

A queue-level retry budget should be shared across all workers for the same operation, not allocated per worker. Store the attempt count and the accumulated estimated cost on the job record, and update them atomically before each attempt. If the job is redelivered, it resumes with the existing budget instead of starting fresh.

A small schema is enough:

alter table generation_jobs
  add column attempts integer not null default 0,
  add column estimated_spend_usd numeric(12, 6) not null default 0,
  add column deadline_at timestamptz not null,
  add column state text not null default 'pending';

Before sending, increment attempts, check deadline_at, and reserve an estimated cost. After the attempt, write the actual or estimated usage. If the budget is exhausted, move the job to failed_budget rather than failed_retryable; the two need different alerts and different human responses.

This also prevents the classic pattern where a poison job is retried forever. A job that fails deterministically should exhaust a small attempt count and land in a dead-letter queue, not consume the shared retry budget for every other job.

A five-minute audit of your current retry path

You can find most of the hidden cost without a major project. Answer these questions for one endpoint:

How many retry layers exist between the user action and the provider? Name them.
Which layer owns the budget, and are the others disabled or configured to defer?
What is the timeout for the first attempt, and where did that number come from?
On timeout, does the client cancel the request or merely stop reading?
Are permanent errors excluded from retries before the backoff logic runs?
Does the retry reuse the same idempotency key and the same parameters?
Is the key generated before the first attempt and persisted across restarts?
Does the job record accumulate cost across attempts, or reset on redelivery?
Can you list every attempt for one logical operation from your logs?
Can you tell whether a failed attempt was billed?

If any answer is "I think so," treat it as "no" until you've checked. Retry behavior is one of those areas where the diagram and the code drift apart quietly, usually in the layer someone added to improve reliability.

What retries cannot fix

Some failures are permanent, and retrying them is pure cost:

400 Bad Request: the request is malformed. The same request will fail the same way.
401 / 403: authentication or authorization. Refresh credentials or fix permissions, don't retry.
404: the model or endpoint doesn't exist. Retrying won't create it.
422: validation failure. Fix the payload.
Context length exceeded: the prompt is too large. Retrying sends the same oversized prompt.
Content policy violation: the same content will be rejected again.
Quota exhausted for the billing period: retries won't add quota.

A useful test: if the request body is byte-for-byte identical, is there any reason to believe the outcome will change? If the answer is no, don't retry. If the answer is yes, the reason should be a transient condition you can name.

Retry, fallback, or fail fast

Retrying is one of three responses to failure. The others are often better.

Strategy	Use when	Cost profile	Risk
Retry same route	Transient error, short outage	Multiplies attempt cost	Retry storm, duplicate work
Fallback to another model	Primary degraded or rate-limited	Similar or lower per attempt	Different quality; schema drift
Shorter prompt / less context	Context too large or latency too high	Lower per attempt	Lost information
Queue and return later	Work is not interactive	Deferred, often cheaper	User waits; state must persist
Fail fast	Permanent error or exhausted budget	One attempt	User sees an error
Degraded deterministic path	Model unavailable, low-risk task	Near zero	Lower quality, clearly labeled

Fallback deserves its own warning. A fallback model may have a different context window, different tool-calling behavior, and a different output style. If the fallback silently produces a lower-quality answer, the retry policy has traded reliability for a quality regression that no error metric will show. Gate fallbacks with the same evals you use for a model upgrade, and label the response so the product can decide whether to show it.

Measure the retry tax

The metrics that matter are not "retries per minute." They are:

Attempts per successful task, overall and by endpoint.
Retry ratio, attempts divided by logical requests.
Billable failure rate, the share of failures that likely incurred cost.
Unknown outcome count, requests whose final state was not reconciled.
Duplicate side-effect count, the number of times a retry caused a second real-world action.
Retry spend, cost attributable to attempts that did not produce the kept result.
Time-to-success, including backoff and queue time.
Cache write rate on retries, because a retry that misses the prompt cache can pay the write premium again.

Emit one structured event per attempt, not just per request:

{
  "operation_id": "gen_01J9F2K7",
  "request_id": "req_8f2c",
  "attempt": 2,
  "endpoint": "responses.create",
  "model": "model-name",
  "status": "error",
  "error_class": "timeout",
  "http_status": null,
  "estimated_cost_usd": 0.018,
  "prompt_tokens": 18400,
  "completion_tokens": null,
  "retry_after_s": null,
  "elapsed_ms": 30000
}

Then aggregate by operation_id. The difference between the sum of attempt costs and the cost of the successful attempt is the retry tax. If you can't compute that, you can't manage it.

🔍 Why this matters: A retry policy that improves the success rate from 97 to 99 percent can still double the cost per success. Without attempt-level accounting, the reliability win looks free.

A policy worth adopting

If you take one thing from this article, make it this: retries should be budgeted like money, because they are money.

A practical default for a user-facing AI feature:

Classify errors before retrying. Never retry permanent client errors.
Give exactly one layer ownership of the retry budget.
Cap attempts, total spend, and wall-clock deadline per operation.
Use full jitter and respect Retry-After.
Cancel timed-out requests explicitly; treat the outcome as unknown.
Keep the model call separate from the side effect, and make the side effect idempotent.
Log every attempt with a stable operation ID and estimated cost.
Reconcile unknown outcomes before retrying them.
Prefer fallback or queueing over a third identical attempt.
Review attempts per success and retry spend alongside latency and quality.

The hidden cost of retries isn't a line item on the invoice. It's a tax on every successful task, paid in tokens for attempts nobody saw, in duplicate actions nobody meant to take, and in capacity consumed while a dependency was already failing. Once you measure it, the policy gets simpler: retry the failures that are likely to succeed, pay only for attempts you can account for, and stop treating a second request as a free replay of the first.