A user types a question, hits enter, and watches a spinner. Behind the spinner, the request hits a 429, waits 2.1 seconds, retries, hits another 429, waits 4.4 seconds, retries, gets a partial response, times out at 30 seconds, and fails. The user sees "Something went wrong. Please try again." They try again. The cycle repeats.
The retry logic was textbook. Exponential backoff, jitter, a max of four attempts. It just happened not to know that the user's patience expired at eight seconds, and that the retries it considered free were consuming the only resource that mattered.
Rate limiting and backoff are usually discussed as server-reliability topics. In AI products they are UX topics, because the dominant cost of a rate limit is not a failed request — it is a person waiting for something that will never arrive in time.
AI rate limits are not web rate limits
Classic API rate limiting counts requests per second. AI providers count something more awkward: often requests per minute, input tokens per minute, and output tokens per minute, enforced per organization and per model. Hitting any one of them produces a 429 even when the others have headroom.
Three consequences follow.
A request consumes budget while it runs. A long generation that takes twenty seconds holds tokens against your per-minute limit for twenty seconds. Concurrency and token throughput interact in ways that pure request-rate limiting does not model, and a burst of long requests can throttle you even at low request volume.
The cost of a retry is not uniform. Retrying a 200-token classification is cheap. Retrying a 40,000-token document analysis is expensive in tokens, in latency, and in the probability that it fails again for the same reason. Retry policy should be per-endpoint, not global.
Rate limits are shared. One noisy feature — a nightly batch job, a re-indexing run, an internal tool someone left in a loop — can starve the user-facing product, because they draw on the same organizational quota. This is the AI equivalent of a batch job saturating the primary database, and it is just as common.
If you have not looked at your provider's rate-limit headers recently, do it before tuning anything. OpenAI's documentation, for example, lists Retry-After alongside x-ratelimit-limit-requests and project-scoped token reset headers, and notes that Retry-After may appear on both 429 responses and 503 responses caused by temporary overload (OpenAI rate limits). Those headers are the only authoritative signal you have about when the window reopens.
What a 429 actually tells you
A 429 is not one condition. It is at least four, and they need different responses.
| Cause | Signal | Right response |
|---|---|---|
| Per-minute request limit | x-ratelimit-remaining-requests: 0 |
Short wait, then retry; smooth the rate |
| Per-minute token limit | token reset header | Wait for the window, or reduce prompt size |
| Concurrency limit | immediate 429 on new requests | Queue with limited parallelism |
| Hard quota / billing | 429 or 403 with a quota message | Do not retry; alert a human |
The last row is the one that burns money and time. A quota exhaustion looks like a rate limit to a naive client, so it retries with backoff for thirty seconds and then fails anyway. Providers generally distinguish these in the error body; read it, and classify before deciding to retry.
Retry-After deserves specific treatment. It is a minimum wait, not a suggested one. If the header says 12 seconds and your backoff says 2, waiting 2 seconds produces another 429 and burns an attempt. The correct composition is: take the larger of your computed backoff and the header value, then add jitter on top so a fleet of clients does not synchronize.
RFC 9110 defines Retry-After for exactly this purpose and allows either a delay in seconds or an HTTP date (RFC 9110 §10.2.3). Handle both forms. Some proxies and gateways emit the date form, and a parser that assumes an integer will silently produce a NaN delay and an immediate retry loop.
Backoff math that respects a deadline
The standard pattern is exponential backoff with full jitter. That is the right starting point, and it is incomplete for interactive products, because it optimizes for eventual success and ignores the wall clock.
A retry loop in a user-facing request should be built around a deadline, not an attempt count. The deadline comes from the product: how long is this interaction allowed to take before the answer stops being useful?
import asyncio, random, time
import httpx
RETRYABLE = {429, 500, 502, 503, 504}
async def call_with_deadline(
client: httpx.AsyncClient,
request: httpx.Request,
*,
deadline_s: float, # total budget for this user action
max_attempts: int = 4,
base_s: float = 0.5,
cap_s: float = 8.0,
) -> httpx.Response:
started = time.monotonic()
attempt = 0
last_exc: Exception | None = None
while attempt < max_attempts:
remaining = deadline_s - (time.monotonic() - started)
if remaining <= 0.25:
break
attempt += 1
try:
resp = await client.send(request, timeout=httpx.Timeout(remaining))
except (httpx.TimeoutException, httpx.TransportError) as exc:
last_exc = exc
resp = None
if resp is not None and resp.status_code not in RETRYABLE:
return resp
# honor Retry-After as a floor, if present and parseable
floor = 0.0
if resp is not None:
ra = resp.headers.get("retry-after")
if ra and ra.isdigit():
floor = float(ra)
backoff = min(cap_s, base_s * (2 ** (attempt - 1)))
delay = max(floor, random.uniform(0, backoff))
if delay >= deadline_s - (time.monotonic() - started):
break
await asyncio.sleep(delay)
if last_exc:
raise last_exc
raise TimeoutError(f"deadline of {deadline_s}s exceeded after {attempt} attempts")
Three details in that loop matter more than the backoff formula.
The timeout is the remaining budget, not a fixed per-attempt value. A fixed 30-second timeout on each attempt lets four attempts consume two minutes while every individual timeout looks reasonable.
The loop refuses to sleep past the deadline. Sleeping for six seconds when only two remain is worse than failing now, because it converts a fast failure into a slow one with the same outcome.
There is a final-attempt guard. If delay would exceed the remaining budget, stop. This is where most "we retry up to four times" implementations accidentally take 40 seconds.
For background jobs the deadline can be generous or absent, and the same function works with a large value. The difference between interactive and batch is a parameter, not a separate code path — which matters, because duplicated retry logic is how the two drift apart.
The conflict nobody names: retry budget versus latency budget
Retries are a reliability technique that spends latency to buy success. That is a good trade when the latency is invisible and a bad trade when a human is watching.
Consider a 95th-percentile request. It hits a 429 on attempt one. Backoff of 1.7 seconds, retry succeeds. Total time: 4.2 seconds. Fine. Now consider a request that hits 429 on attempts one, two, and three. Waits of 1.7, 3.4, and 6.9 seconds. Total: 16 seconds plus generation time. The request eventually succeeds, and the user has already clicked away.
Your retry logic reports a 99.4 percent success rate. Your users report that the product is slow and unreliable. Both are true, and the metric that reconciles them is success within the patience budget, not success eventually.
Instrument it explicitly:
p_success_within_budget = count(success and elapsed <= budget) / count(all requests)
For an interactive endpoint with a 10-second budget, a system that succeeds 99 percent of the time but only 91 percent within budget should be treated as a 91 percent system. The remaining 8 points are failures with extra steps.
How long will users actually wait?
Patience is not a single number. It depends on what the user asked for, what they can do meanwhile, and what they have already invested.
The ranges below are product judgment, not measured constants, and they should be calibrated against your own abandonment data. But they are a reasonable starting point for setting deadlines.
| Interaction | Comfortable | Tolerable with feedback | Abandonment zone |
|---|---|---|---|
| Autocomplete / inline suggestion | < 300 ms | 1 s | > 2 s |
| Chat reply, first token | < 1.5 s | 3 s | > 6 s |
| Chat reply, full answer | < 8 s | 20 s | > 30 s |
| Document summary (user initiated, has a progress view) | < 15 s | 60 s | > 2 min |
| Report generation (user can leave the page) | any | any, with notification | — |
| Agent task with visible steps | < 2 s per step | 10 s per step | step appears frozen |
Two patterns in that table are worth internalizing.
Streaming changes the patience curve. Users tolerate a 12-second answer that starts writing at 1.2 seconds far better than a 6-second answer that appears all at once. First-token latency is a separate budget from total latency, and it is usually the one that determines whether the user stays.
Visible progress changes it more. A multi-step agent that shows "searching docs… reading 4 results… drafting" buys far more patience than a spinner, even at identical total duration. What users punish is not slowness; it is unexplained slowness.
What to do instead of retrying
When the deadline is short, retrying is often the wrong tool. The alternatives are better because they deliver something now.
Queue with position. If the request cannot be served immediately, admit it to a queue and say where it is. "You're 3rd in line, about 20 seconds" is a better experience than three invisible retries and a failure, even though it is slower.
Degrade the model. A smaller, faster model that answers in two seconds beats a frontier model that times out. This requires having defined which tasks tolerate the quality drop — summarization of a short thread usually does, contract analysis usually does not. Route by task, and tell the user which mode they got when it matters.
Return a partial answer. For long-form generation, deliver what exists and offer to continue. Users prefer an incomplete answer with a "continue" button to no answer at all.
Defer to async. For anything over the patience threshold, hand off to a background job and notify. The important design detail is that the user must be able to leave the page and come back; a job that only delivers into an open tab has not actually solved the latency problem.
Fail fast and specifically. If the provider is down, saying so in 1.5 seconds is better than discovering it in 25. A fast, honest error lets the user change their approach; a slow timeout does not.
⚠️ Gotcha: a queue in front of a rate-limited provider converts a 429 into an unbounded wait unless you cap the queue. Set a maximum wait and shed beyond it. An unbounded queue during an incident is how you get users waiting four minutes for errors.
Hedging: duplicate requests are a cost decision
The latency tail of an LLM call is long and mostly unpredictable. One response comes back in 900 ms; the next, same prompt, takes 6 seconds. Hedging — sending a second request after a short delay and taking whichever finishes first — cuts the tail dramatically.
It also doubles your token spend on the requests that hedge, and it consumes rate-limit budget you may not have. Hedging during a rate-limit incident makes the incident worse, because the duplicate request competes for the same exhausted window.
A defensible policy: hedge only above a latency threshold (say, the 90th percentile of recent successful calls), only for endpoints where the spend is small relative to the user value, and never while the client is seeing 429s. The hedge decision is a function of both latency and headroom, and most implementations only look at latency.
Abandonment and cancellation are part of the retry design
When a user closes the tab, the request keeps running on the server, keeps consuming tokens, and may still write to a cache or a database. A retry loop that ignores cancellation will happily finish four attempts for a user who left after two seconds.
Propagate the cancellation signal through the whole chain, including the retry loop's sleep. In most async runtimes that is a single check inside the loop and a cancellable sleep rather than a bare sleep(). On the client, AbortController and its equivalents exist for exactly this (MDN: AbortController).
The cost is not theoretical at scale. If 15 percent of long requests are abandoned at 10 seconds and your retry policy keeps working until 45, you are paying for a meaningful share of work nobody will read.
Nested retries are how one failure becomes sixty-four
The most common cause of a self-inflicted rate-limit incident is not a traffic spike. It is retries multiplying across layers that each believe they are being conservative.
A typical stack has three: the provider SDK retries twice by default, your application retries four times, and the gateway or service mesh retries twice on 5xx. If each layer is independent, a single user action can produce up to sixteen provider calls, and under synchronized load the retry waves amplify each other. Add a client-side retry in the browser and the arithmetic gets worse.
The fix is to establish exactly one retry owner per call path, and to make the other layers passive.
- Disable SDK retries when you implement retries above it. OpenAI's documentation says this explicitly: if you manage retries in your application, turn off SDK retries or account for them in your limits so nested loops do not multiply requests.
- Configure the mesh to not retry AI routes. A mesh retry on a 429 is doubly wrong: it ignores the
Retry-Afteryour application would have honored, and it multiplies load precisely when the upstream asked for less. - Ban browser retries on generation endpoints. A user clicking "try again" is a legitimate new request; a
fetchwrapper retrying in the background is not.
If you keep SDK retries because they are convenient, then your application layer should not retry at all — it should fail, classify, and let the queue or the user decide. Either arrangement is defensible. Both at once is how you turn a ten-second rate-limit window into a two-minute outage.
Rate limiting your own users, before the provider does it for you
If you wait for the provider's 429 to tell you that you are over budget, you have already lost control of the experience. You are now sharing a global limit across all tenants, and the noisiest one wins.
Enforce your own limits first, with a token bucket per tenant and per endpoint, sized from the provider quota divided by expected concurrency. Two design choices make this materially better than a naive request counter.
Meter in tokens, not requests. Estimate the prompt tokens before the call and reserve them; reconcile against the actual usage afterward. A tenant sending 40,000-token documents at ten requests per minute is consuming a very different share of the quota than one sending 400-token messages at the same rate.
Return a real Retry-After on your own 429s. If the client is your frontend, it can render "try again in 8 seconds" instead of a generic error. If the client is a customer's integration, they can implement correct backoff because you told them how. An undocumented 429 with no header guarantees that every client invents its own retry policy, and most will invent a bad one.
There is a fairness question underneath this that is easy to postpone and expensive to retrofit: should one tenant be able to consume 80 percent of the quota because they are the largest? For internal products the answer is usually yes, with a ceiling. For a multi-tenant SaaS product it usually is not, and the limit needs to be per-tenant with headroom reserved for the rest. Deciding this after launch means changing behavior that customers have already built around.
Timeouts that match the actual task
A single global timeout is a bug waiting to happen. The timeout on a classification call and the timeout on a 30-page summarization should not be the same number, and neither should be a round default inherited from an HTTP library.
Set timeouts from measured distributions, with three separate values:
- Connect timeout: short, 2–5 seconds. If the TCP handshake is slow, the network is the problem and retrying elsewhere is the right move.
- Read/idle timeout: how long the stream may go silent before you give up. For streaming responses this is the one that matters, and it should be well above your observed inter-token gap, not above your total duration.
- Total deadline: the patience budget from the product. This is enforced by your retry loop, not by the HTTP client.
The read timeout is where streaming systems most often get it wrong. A 30-second total timeout kills a legitimate long generation that is streaming steadily; a 30-second idle timeout correctly kills a connection that has gone quiet. Those are different failures and deserve different treatment: the first should have been a queue, the second is a genuine network or provider problem.
🔍 Why this matters: a timeout is a statement about what "too slow" means for this specific task. Copying it from another endpoint is a guess dressed up as a configuration.
Retry budgets: the system-wide cap
Per-request attempt limits do not bound the total load that retries add. If every request is allowed four attempts and a dependency degrades, you can end up sending four times your normal traffic at a service that is already struggling. Each individual client is behaving correctly; the fleet is not.
A retry budget fixes this at the fleet level: allow retries only while they stay below a fixed ratio of successful calls. Something like ten percent is a common starting point — for every hundred successful requests, up to ten retries are permitted. When the dependency degrades and the ratio is exceeded, retries stop, and requests fail fast instead of piling on.
The mechanism is a shared counter with a short window, not a per-client counter, and it has to be cheap enough to check on every attempt. A Redis counter with a one-minute TTL is usually enough. The important property is that the budget degrades gracefully: under normal conditions retries absorb blips, and under real stress they stop amplifying the problem.
Budgets pair naturally with load shedding. When the budget is exhausted, shed the lowest-value work first — background re-indexing, precomputation, anything a user is not currently waiting for — and protect the interactive path. That ordering has to be decided in advance, because during an incident there is no time to debate which traffic matters.
🚨 Production warning: a retry budget with no alert is invisible. If it activates and nobody notices, you have silently converted a retry policy into a failure policy. Emit a metric and page on sustained activation.
Metrics that connect rate limits to the product
Most rate-limit dashboards are built for infrastructure: 429 rate, retry count, provider error codes. Those are necessary and insufficient. Add these:
- Success within patience budget, per endpoint, per interaction type.
- First-token latency p50/p95, separate from total latency.
- Abandonment rate and the median time-to-abandon. If it is close to your deadline, the deadline is wrong.
- Retry amplification: total provider requests divided by user requests. Above roughly 1.2 during normal operation, something is retrying more than it should.
- Rate-limit share of failures: how often a user-visible failure traces to a 429, as opposed to a model or application error.
- Queue wait p95, if you queue at all, and the share of queued requests that never get viewed.
The last one is easy to forget and it is the honest measure of whether your queue is helping or just hiding. A queue that delivers 40 percent of its answers after the user has gone is a latency problem with a nicer chart.
Tuning the knobs in a sane order
When the product feels slow and the 429s are frequent, resist the urge to start with backoff parameters. The order that produces results:
- Measure the patience budget per endpoint. Talk to the product owner. "How long can this take before it's useless?" is a question with an answer, and it is the input to everything else.
- Separate interactive from batch traffic into different rate-limit pools. If you cannot, one nightly job will keep ruining your afternoons.
- Reduce token consumption per request before adding concurrency. Caching the prefix, trimming retrieved context, and capping output length reduce pressure on the exact limits that are biting.
- Set deadlines and propagate them. Replace attempt counts with time budgets in the interactive path.
- Honor
Retry-Afteras a floor and add jitter. This is a small change with an outsized effect during incidents. - Only then add hedging, and only where the numbers justify the spend.
- Add degraded modes for the endpoints where a fast, worse answer is acceptable.
Each step makes the next one cheaper. Teams that start at step 6 buy themselves more traffic during an incident and a larger bill afterward.
A decision framework
For any AI endpoint, answer these four questions and the configuration falls out.
How long can this take before the answer stops being useful? That is the deadline. If the honest answer is "there isn't one", it is a batch job and belongs in a queue, not a request.
What is the cost of a failed attempt relative to a slow success? Cheap retries on cheap requests are fine. Expensive retries on expensive requests should be rare, and only after the second failure looks transient.
Can I show the user something while waiting? If yes, invest there before investing in retry tuning. Perceived latency responds to design more than to infrastructure.
What happens to the request when the user leaves? If it keeps running, cancel it. If it cannot be cancelled, keep it short.
Rate limiting is the part of an AI product where infrastructure decisions become user experience decisions, and the two are usually made by different people. The teams that get this right put the patience budget in the same document as the rate-limit configuration, so that someone tuning the backoff has to look at the number that says how long a human will wait.