Designing AI features that degrade gracefully when the API is slow

The first symptom is rarely an error. It is a blinking cursor.

A support agent clicks Summarize this ticket. The summary usually appears in about a second. Today the cursor sits there for nine seconds, then twelve. The agent assumes the click did not register and clicks again. Two requests are now in flight. The model eventually answers both, and the ticket ends up with two slightly different drafts. Nothing in the provider dashboard turned red. Every request returned HTTP 200. The feature was still broken.

That is the peculiar failure mode of AI products: the dependency can be healthy, the status page can be green, and the feature can still be unusable because the useful part of the response arrived after the human gave up. Graceful degradation is not only about handling 500s and 429s. It is about deciding what the product does with the two seconds, five seconds, or thirty seconds before an answer exists, and what it does when the answer only half exists.

For the rest of this article, graceful degradation means three things. The user can still complete the task, or understands precisely why they cannot. The system does not amplify the slowdown by retrying blindly. And any lower-quality output is labeled as lower quality rather than quietly presented as the real thing. Everything below is in service of those three properties.

Slow is a state, not an exception

Most applications treat latency as a scalar in a dashboard. AI features need to treat it as a state machine. A request can be queued, connecting, waiting for the first token, streaming, stalled mid-stream, partially complete, or complete but stale. Each state has a different correct action.

The reason is structural. A generated answer is produced token by token, so the time to the first useful token and the time to the last token are different products. OpenAI's latency guide makes this distinction directly: streaming does not reduce the total time to generate all tokens, but it reduces the time before the user sees progress. That single sentence should change how you instrument the feature. If you only measure total request duration, you will miss the outage your users actually experience.

There is also a taxonomy problem. "The API is slow" can mean the provider is queueing your request, your own backend is queueing it, the network path is losing packets, the model is generating a long answer, your post-processing is slow, or your client is waiting for a tool call that will never finish. These have different remedies. A retry helps with a transient network reset. It makes a capacity shortage worse. A cheaper model helps with a slow, oversized model. It does nothing for a broken TLS handshake.

So start with the budget, then classify the delay, then choose the degradation.

Put a number on every phase

A latency budget is a product decision expressed as a deadline. It is not the same thing as an HTTP timeout. The timeout belongs to one attempt; the budget belongs to the whole user-visible operation, including retrieval, prompt assembly, queue wait, retries, streaming, validation, and rendering.

Consider a hypothetical support-reply feature with a twelve-second interaction budget. These numbers are illustrative, not universal:

Phase	Budget	Why it exists
Admission and queue wait	250 ms	Beyond this, the user needs a queued state
Retrieval and policy lookup	600 ms	The answer is only as good as its context
Prompt assembly and safety pre-checks	100 ms	Local work should not consume the provider budget
Time to first content token	2,500 ms	The point where a blank screen starts feeling broken
Streaming the full answer	7,000 ms	Generation is usually the dominant phase
Validation, persistence, render	500 ms	The answer is not done until it is safe to show
Reserve	1,050 ms	Room for one bounded fallback or a clean failure

The reserve is the part teams forget. If the primary attempt is allowed to consume the entire budget, there is no time left to do anything graceful. A budget without a reserve produces a timeout, not a degradation strategy.

Budgets should be inherited, not re-created at every layer. Pass a deadline through the call graph and derive each phase's allowance from the remaining time. A small deadline object is enough:

export class Budget {
  private readonly deadline: number;

  constructor(totalMs: number, readonly startedAt = Date.now()) {
    this.deadline = startedAt + totalMs;
  }

  remaining(): number {
    return Math.max(0, this.deadline - Date.now());
  }

  expired(): boolean {
    return this.remaining() === 0;
  }

  cap(ms: number): number {
    return Math.min(ms, this.remaining());
  }
}

The rule that matters is cap, not the class. Every network call, retry sleep, queue wait, and fallback gets budget.cap(desiredTimeout). If the result is below a viable minimum, do not start the call. Go straight to the degraded path.

This is also where you decide what "viable" means. A 40 ms retrieval call is not viable; it will almost certainly time out and waste the remaining budget. A 900 ms streaming attempt may be viable if the user only needs the first paragraph. The threshold is task-specific.

A timeout taxonomy that matches reality

One timeout is never enough for an AI request. At minimum, distinguish these:

Timeout	Starts when	Protects against	What happens if it is missing
DNS/TCP/TLS connect	Socket creation	Dead endpoints, bad network paths	The request hangs before the provider ever sees it
Request headers / first byte	Request sent	A provider that accepted the connection but never responds	The UI waits on a request that has not started generating
First content token	Stream opened	A model that is queueing or thinking for too long	The user sees an empty stream indefinitely
Inter-chunk idle	Last content delta	A stalled stream after a successful start	A partial answer sits frozen with no error
Per-attempt total	Attempt start	One pathological attempt consuming the operation	Retries start with no budget left
Operation deadline	User action	The sum of all attempts, retries, and fallbacks	Each layer behaves reasonably while the whole request fails late
Queue wait	Job enqueued	Work that will never be picked up in time	Users wait on a queue that is silently backing up
Client-visible deadline	Render start	A browser or mobile client that waits forever	The server gives up but the user does not know

Two of these deserve special attention in AI systems: first content token and inter-chunk idle.

The first content token is not the same as the first SSE event. Providers emit lifecycle events before any text. With the OpenAI Responses API, for example, the stream includes events such as response.created and response.output_text.delta; only the latter carries user-visible text. If your watchdog resets on every event, a provider that sends pings or metadata will look healthy while producing nothing.

Inter-chunk idle is the failure that streaming introduces. A stream can open successfully, send a few tokens, and then stall because of a dropped connection, an overloaded inference worker, or a proxy that has stopped forwarding bytes. A total request timeout may not fire for a long time. You need a separate timer that resets only on meaningful content and aborts when the gap exceeds the budget.

An explicit stream watchdog looks like this:

import OpenAI from "openai";

const client = new OpenAI({ maxRetries: 0 });

export async function streamAnswer(
  input: string,
  budget: Budget,
  onDelta: (text: string) => void,
): Promise<{ text: string; completed: boolean }> {
  const controller = new AbortController();
  let text = "";
  let completed = false;

  const firstTokenTimer = setTimeout(
    () => controller.abort(new Error("first_token_timeout")),
    budget.cap(2_500),
  );

  let idleTimer: NodeJS.Timeout | undefined;
  const resetIdleTimer = () => {
    if (idleTimer) clearTimeout(idleTimer);
    idleTimer = setTimeout(
      () => controller.abort(new Error("stream_idle_timeout")),
      budget.cap(4_000),
    );
  };

  try {
    const stream = await client.responses.create(
      {
        model: "gpt-5.6-terra",
        input,
        stream: true,
      },
      { signal: controller.signal, timeout: budget.cap(20_000) },
    );

    for await (const event of stream) {
      if (event.type !== "response.output_text.delta") continue;
      clearTimeout(firstTokenTimer);
      resetIdleTimer();
      text += event.delta;
      onDelta(event.delta);
    }

    completed = true;
    return { text, completed };
  } catch (error) {
    const reason =
      error instanceof Error ? error.message : "stream_failed";
    if (text.length > 0) {
      return { text, completed: false };
    }
    throw new Error(reason);
  } finally {
    clearTimeout(firstTokenTimer);
    if (idleTimer) clearTimeout(idleTimer);
  }
}

This is deliberately not a complete production client. It assumes Budget is passed in, maxRetries: 0 because retries are handled by an outer policy, and the caller persists the partial text if the stream fails. The important detail is that a partial result is returned rather than discarded. Throwing away 80% of an answer because the last 20% never arrived is a design choice, and usually the wrong one.

Do not confuse the per-attempt timeout with the operation deadline. OpenAI's Node SDK, for example, documents a default request timeout of ten minutes and automatic retries for connection errors plus HTTP 408, 409, 429, and 500-or-higher responses, twice by default. Those defaults are reasonable for a batch job and dangerous for a twelve-second chat. Anthropic's SDK similarly retries transient failures twice by default and honors retry-after. Gemini's Python SDK retries transient errors up to four times with an initial delay around one second and a maximum delay of sixty seconds, according to Google's documentation. If you add your own retry loop on top without disabling SDK retries, you are not adding one retry. You are multiplying them.

Retries are a loan against the budget

A retry is not free. It consumes the user's patience, the provider's capacity, and often your money. It is justified when the failure is transient and the request is safe to repeat. It is harmful when the failure is caused by overload, when the request has side effects, or when the remaining budget cannot accommodate another attempt.

Classify before you retry:

• Retryable: connection resets, DNS failures, 408, 429 with a sane Retry-After, 500, 502, 503, 504, and provider-specific overload errors such as Anthropic's 529.
• Not retryable without a change: 400, 401, 403, 404, 413, and 422. Repeating the same malformed or unauthorized request just burns budget.
• Not retryable at all: payment required, exhausted credits, a spend cap, or a policy refusal. A 429 is ambiguous here. It can mean "slow down" or "you are out of money." Anthropic documents that a tier spend-cap 429 has no retry-after and keeps failing until access resumes. OpenAI exposes billing and spend-limit conditions through specific error codes; Gemini uses 402 for depleted prepaid credits. Read the body and headers, not just the status.

When a Retry-After header exists, treat it as a minimum, not a suggestion. If the value exceeds the remaining budget, do not sleep through the deadline. Fall back or queue immediately. If the header is missing or invalid, use capped exponential backoff with jitter. AWS's SDK guidance describes full jitter as random(0, 1) × min(cap, base × 2^retry), with a twenty-second cap in its standard retry mode; the exact base delay differs by error class. You do not need to copy those constants, but you do need the shape: exponential growth, a ceiling, and randomness.

Jitter is not cosmetic. If a provider has a brief blip and every client retries after the same one-second delay, the retry wave becomes a second incident. Full jitter spreads the wave. It also makes your own logs less periodic and easier to read.

Two budget guards are worth adding:

1. A per-request attempt cap. Three attempts is a common starting point, but the right number depends on the task. A background enrichment job can afford more than an interactive completion.
2. A retry ratio budget. Google's SRE book describes a per-client budget where retries are allowed only while they remain below roughly 10% of requests. The exact percentage is a policy choice; the principle is that retries must not become a large fraction of total traffic during an incident.

The arithmetic of nested retries is brutal. Three layers, each with two retries, can produce twenty-seven attempts for one user action. That is not a hypothetical benchmark; it is just multiplication. Decide which layer owns retries. Usually it is the layer closest to the provider, with the outer layers owning fallback and queueing instead.

Hedged requests: buy tail latency, pay in tokens

Hedging means sending a second copy of a request before the first one fails, then using whichever answer arrives first and cancelling the loser. The canonical treatment is Dean and Barroso's The Tail at Scale, which popularized deferring the second request until the first has been outstanding longer than the expected p95 latency. That threshold keeps the duplicate rate low in normal operation while cutting the tail.

For ordinary RPCs, hedging is a well-understood latency tool. For LLM calls, it is more expensive and more subtle. Duplicate generations consume tokens. Cancelling an HTTP request does not necessarily erase the provider's cost for work already performed; design as if duplicated tokens are billed. Hedging a long-form generation can also produce two different answers with different facts, and choosing "the first one to finish" is not the same as choosing the better one.

That does not make hedging useless. It makes it a targeted tool:

• Hedge short, read-only classification or extraction calls where the output is easy to validate.
• Hedge retrieval or embedding lookups where the result is deterministic for the same input.
• Do not hedge a long, expensive generation unless the task can tolerate either answer and the cost is acceptable.
• Cap the hedge at two in-flight attempts.
• Cancel the loser, and record whether the hedge won.
• Never hedge when the circuit breaker is open or the system is already shedding load. Hedging during overload is just retrying with better branding.

A compact implementation for a non-streaming, idempotent call:

export async function hedged<T>(
  start: (signal: AbortSignal) => Promise<T>,
  hedgeAfterMs: number,
  budget: Budget,
): Promise<T> {
  const controllers = [new AbortController(), new AbortController()];
  const timers: NodeJS.Timeout[] = [];

  return new Promise<T>((resolve, reject) => {
    let settled = false;
    let failures = 0;

    const finish = (fn: () => void) => {
      if (settled) return;
      settled = true;
      timers.forEach(clearTimeout);
      controllers.forEach((c) => c.abort());
      fn();
    };

    const launch = (index: number) => {
      start(controllers[index].signal).then(
        (value) => finish(() => resolve(value)),
        (error) => {
          failures += 1;
          if (failures === 2) finish(() => reject(error));
        },
      );
    };

    launch(0);
    timers.push(setTimeout(() => launch(1), budget.cap(hedgeAfterMs)));
  });
}

The code assumes start is safe to run twice and that the provider will stop work when the signal aborts. Both assumptions need verification for your provider and endpoint. If they do not hold, hedging is not a latency optimization; it is a duplicate-work generator.

Streaming is a perception fix, not a capacity fix

Streaming is the highest-leverage degradation technique because it changes the shape of waiting. The user sees the answer being written instead of staring at a spinner. But streaming does not make the model faster. It moves the uncertainty from "will I get an answer?" to "will this answer finish?"

That shift creates new obligations. You need a first-token deadline, an idle-chunk deadline, a way to persist partial text, and a policy for what happens when the stream dies after the 200 response. Anthropic's documentation is explicit that an error can occur after a streaming response has started, and that such errors do not follow the normal HTTP error path; they arrive as error events in the stream. OpenAI's streaming guide likewise lists an error event among the lifecycle events to handle. A try/catch around the initial request is not enough.

There is also a continuity problem. If the stream fails after 60% of a paragraph, you have three options: discard it, show it as partial, or ask the model to continue. Continuation is provider- and model-specific. Anthropic's docs describe capturing the partial response and, for Claude 4.6 and later, sending a user message that contains the partial text and an instruction to continue; for earlier models the same capture-and-resume idea uses the partial response as the start of an assistant message. Do not assume every provider supports the same trick, and do not silently stitch a continuation from a different model onto the original text. If the model changes, mark the boundary.

The most important streaming rule is simple: once you have shown text, you cannot un-show it. A retry that starts from scratch after the user has read half an answer is usually worse than an honest partial state. If the content is safety-sensitive, buffer until validation; if it is a draft, stream it and label it as a draft.

That tension leads directly to the next design question: what exactly is a partial result allowed to contain?

Partial results need a contract

"Partial" is not one thing. It helps to separate three layers:

• Syntactic partial: the stream stopped mid-sentence or mid-JSON.
• Semantic partial: the answer is complete prose but is missing a required fact, source, or tool result.
• Business partial: the system can complete only part of the workflow, such as showing an order summary without the live shipping status.

Each needs a different representation. A truncated sentence can be shown as a draft. A missing fact should be named, not guessed. A missing workflow step needs an action the user can take.

For structured output, the worst pattern is to parse a partial object, fill defaults for the missing fields, and pass it downstream as if it were complete. That turns a latency problem into a data-integrity problem. A better pattern is an explicit envelope:

{
  "status": "partial",
  "answer": "The ticket is about a delayed replacement, and the customer has already provided the order number.",
  "missing": ["refund_eligibility"],
  "as_of": "2026-10-22T14:03:11Z",
  "next_action": "Check refund eligibility in the orders tool"
}

Downstream code can branch on status. The UI can show the answer with a visible "one check did not finish" note. The user can retry just the missing piece if the product supports it. Nothing silently invents refund_eligibility: false.

The same idea applies to retrieval-augmented answers. If retrieval times out, you have three honest options: answer from the model's general knowledge with a clear disclaimer, answer only from the sources that did load and cite them, or decline and offer a retry. What you must not do is generate a confident answer and attach citations to documents the retrieval step never returned. Partial retrieval should reduce the scope of the claim, not the honesty of the citation.

A useful implementation detail is to validate progressively. If the output is JSON, parse at block boundaries or use a streaming JSON parser, but only mark the object complete after schema validation. If validation fails, you can spend one cheap repair attempt if the budget allows; otherwise return the partial envelope. The repair attempt is another model call, so it belongs inside the same deadline, not outside it.

Stale cache: useful, but never invisible

Caching is the oldest degradation strategy, and AI features have several distinct caches: exact response caches, retrieval caches, embedding caches, prompt-prefix caches, and semantic caches that return a previous answer for a sufficiently similar question. They have very different risk profiles.

An exact response cache keyed by the full normalized request, model, prompt version, tool configuration, tenant, and locale is relatively safe for deterministic tasks. A semantic cache is a gamble: two questions can look similar and require different answers, especially around dates, account state, and negations. Use it only where a wrong match is cheap to detect.

The cache key is part of the correctness model. Include everything that can change the answer:

type CacheKeyInput = {
  tenantId: string;
  userId: string;
  promptVersion: string;
  model: string;
  locale: string;
  tools: string[];
  normalizedInput: string;
};

If the prompt version is not in the key, a deploy can serve answers generated by an older policy. If the tenant is not in the key, you can leak one customer's data to another. If the tool list is not in the key, an answer generated without access to a tool may be served to a request that expects the tool's result.

For degradation, the interesting mode is stale-while-revalidate. Serve a cached answer immediately, label it with its timestamp, and refresh it in the background. This is appropriate for documentation summaries, policy explanations, and other slow-moving content. It is not appropriate for account balances, medical guidance, legal deadlines, or anything where "yesterday's answer" is a different answer.

Make staleness explicit in both the API and the UI:

type CachedAnswer = {
  text: string;
  createdAt: string;
  fresh: boolean;
  staleByMs: number;
};

The UI can then render "Based on information from 22 October, 09:14" instead of pretending the answer is current. That label is not a legal disclaimer bolted onto the bottom; it is part of the product's truthfulness.

Three operational details matter:

1. Bound the staleness. A cache entry should have a hard maximum age per content class, not just a TTL. Serving a week-old policy answer because the origin is down is a business decision, not a cache default.
2. Single-flight the refresh. When a popular key expires, dozens of requests will try to regenerate it at once. Use a per-key lock or a request coalescer so one request refreshes and the rest wait or receive the stale value.
3. Jitter the TTL. If thousands of entries were written at deploy time, they will expire together. A small random offset prevents a cache-expiry traffic wave.

Also decide what not to cache. Prompts containing personal data, tool outputs with access-controlled fields, and anything that varies by authorization scope should be partitioned or excluded. A cache that ignores authorization is an access-control bug with a latency benefit.

Cheaper-model fallback is a capability decision

The obvious fallback for a slow or unavailable model is a smaller, faster, cheaper one. The non-obvious part is that "cheaper" is not the same as "equivalent." A smaller model may follow the format but miss a nuance, or produce a plausible answer with a worse calibration. Falling back to it without evaluation is how a latency incident becomes a quality incident.

Treat models as a ladder with explicit contracts:

Tier	Use it for	Contract
Deep	Ambiguous reasoning, long context, high-stakes synthesis	Highest quality, highest latency and cost
Balanced	Routine drafting, extraction with clear schema	Acceptable quality on the evaluated task set
Fast	Classification, routing, short rewrites, simple retrieval answers	Low latency, narrow scope, validated output
Deterministic	Known intents, policy responses, status links	No model call at all

The ladder is only meaningful if you have evaluated each rung on your own examples. A public benchmark does not tell you whether the fast model can handle your refund policy's edge cases. Build a small evaluation set from production-shaped inputs, including the ugly ones, and record quality, format compliance, refusal behavior, and latency for each tier. Then define which routes may downgrade and which may not.

A router can be simple:

type Route = "support_reply" | "ticket_summary" | "intent_classification";

type ModelChoice = {
  model: string;
  tier: "deep" | "balanced" | "fast";
  maxOutputTokens: number;
};

export function chooseModel(
  route: Route,
  budget: Budget,
  health: { deepAvailable: boolean; balancedAvailable: boolean },
): ModelChoice {
  if (route === "intent_classification") {
    return { model: "gpt-5.6-luna", tier: "fast", maxOutputTokens: 128 };
  }

  if (budget.remaining() < 4_000 || !health.deepAvailable) {
    return { model: "gpt-5.6-terra", tier: "balanced", maxOutputTokens: 400 };
  }

  return { model: "gpt-6-astra", tier: "deep", maxOutputTokens: 800 };
}

The model identifiers here are examples from current provider documentation, not a claim that these specific models are interchangeable. The logic is the point: route by task, remaining budget, and observed health. Do not let a generic "fallback model" environment variable silently apply to every route. A downgrade that is safe for classification can be unacceptable for a medical or financial summary.

Provider capacity is often per model, which makes model switching a real availability tool rather than only a cost tool. Anthropic's own troubleshooting guidance for 529 overload suggests switching models, and Gemini's error documentation suggests temporarily switching from a larger model to a smaller one. If you operate multiple providers, the same idea applies across vendors, but the prompts and tool schemas must be portable enough to make the switch credible. That is an architectural investment, not a line of code.

Finally, make the downgrade observable. Record the requested tier, the served tier, and the reason. If the balanced model serves 30% of traffic for a week because a health check is flapping, you want to know before your quality metrics drift.

Deterministic fallback: the floor of the system

The most reliable fallback does not call a model. It is a template, a rule, a search result, a direct link, or a human handoff. This is the path that works when the provider is down, the budget is gone, the breaker is open, and the queue is full.

Deterministic fallback is not the same as a generic error page. "Something went wrong" is a failure state, not a degradation strategy. A useful deterministic fallback answers a narrower question with certainty:

• For order status: show the last known status, the carrier link, and the timestamp.
• For refund policy: show the policy text and a prefilled request form.
• For password reset: link directly to the reset flow.
• For an unknown intent: offer a human handoff and preserve the user's message.
• For a summary task: show the raw ticket with the most recent messages expanded instead of a generated summary.

The design work is choosing the top intents and writing the fallback content before the incident. That content should be reviewed by the same people who review the AI prompt, because it is now part of the product's voice. Keep it short, specific, and actionable. Do not apologize more than once. Do not claim the system will "try again later" unless it actually will.

A deterministic path also gives you a safe default for high-stakes routes. If the task is "compute a refund amount," the fallback should be "show the policy and hand off," not "generate a number with a smaller model." The cheaper model is still a generative system; the deterministic path is the one with a bounded failure mode.

The trade-off is coverage. Templates only handle known situations. That is why they belong at the bottom of a ladder, not as the only strategy. But a system with a good floor can afford to be more aggressive about timing out the upper layers, because the floor is trustworthy.

Circuit breakers: stop paying for a dependency that is failing

A circuit breaker converts repeated failures into fast, local rejections. In the classic model, the breaker is closed while calls succeed, opens after a threshold, rejects calls immediately while open, and moves to half-open after a cool-down to test one request. If the probe succeeds, it closes; if it fails, it opens again.

For AI systems, put breakers around the dependency you actually need to isolate: a provider, a model, a region, or a specific route. A single global "AI is down" breaker is too coarse. The deep model may be overloaded while the fast model is healthy; one provider's embeddings endpoint may be failing while its chat endpoint is fine.

Thresholds need a minimum volume. "Three failures in a row" is meaningless at low traffic and too sensitive at high traffic. A common shape is: open when the error rate exceeds a threshold over a rolling window with at least N requests, or when consecutive infrastructure failures exceed a smaller cap. The exact numbers are workload-specific. What matters is that you only count failures the breaker can actually fix: timeouts, connection errors, 5xx, and provider overload. Do not count 400s caused by bad user input, 401s caused by a bad key, or 413s caused by an oversized request. Those need different fixes.

A small per-instance breaker:

type State = "closed" | "open" | "half_open";

export class Breaker {
  private state: State = "closed";
  private failures = 0;
  private openedAt = 0;
  private probing = false;

  constructor(
    private readonly failureThreshold = 5,
    private readonly coolDownMs = 15_000,
  ) {}

  allow(now = Date.now()): boolean {
    if (this.state === "closed") return true;

    if (this.state === "open") {
      if (now - this.openedAt < this.coolDownMs) return false;
      this.state = "half_open";
    }

    if (this.probing) return false;
    this.probing = true;
    return true;
  }

  recordSuccess(): void {
    this.failures = 0;
    this.probing = false;
    this.state = "closed";
  }

  recordFailure(now = Date.now()): void {
    this.probing = false;
    this.failures += 1;
    if (this.state === "half_open" || this.failures >= this.failureThreshold) {
      this.state = "open";
      this.openedAt = now;
    }
  }
}

This is intentionally minimal. In a multi-instance service, an in-process breaker only protects one process; a fleet-wide breaker needs shared state or a gateway that enforces limits centrally. Envoy, for example, enforces circuit-breaking limits at the network level per upstream cluster, including maximum connections, pending requests, active requests, and active retries. That is a different layer of protection, and the two can coexist: the gateway protects the fleet, while the application breaker protects a specific route or model.

The interaction with retries is important. When the breaker is open, do not retry. When it is half-open, allow one probe, not a burst. When it closes, ramp traffic back gradually rather than releasing the whole backlog at once. Recovery traffic can be as damaging as the original failure.

And make the breaker's state visible. A breaker that opens silently looks exactly like a provider outage from the user's perspective, but the remediation is different. Emit a metric and a log line when it transitions, and include the state in the request's degradation reason.

Load shedding is a capacity policy, not an error

When demand exceeds capacity, something has to be dropped. The only question is whether you choose what to drop or let the queue choose for you. Load shedding is the deliberate version: reject low-value work early so high-value work can still meet its budget.

Google's SRE chapter on handling overload describes criticality classes, from CRITICAL_PLUS down to SHEDDABLE, and rejects lower criticalities first as utilization rises. It also describes client-side adaptive throttling, where clients cap their own outgoing traffic when the backend is rejecting a large fraction of requests. Both ideas translate directly to AI features.

A practical priority scheme:

Class	Examples	Behavior under pressure
Interactive, user-blocking	Chat reply, inline completion, form assist	Admit first; degrade model tier before dropping
Interactive, non-blocking	Suggested tags, related questions	Shed before the blocking class
Background, user-initiated	Long document summary, batch rewrite	Queue with a longer deadline
Background, system-initiated	Embedding refresh, cache warming, analytics enrichment	Pause entirely

Admission control should happen before you spend money on retrieval or prompt assembly. If the request cannot be admitted, return a fast, structured rejection with a Retry-After value and a queue option. Do not accept the work, let it sit in memory, and time out later.

For AI specifically, concurrency limits should be token-aware, not just request-aware. One request with a 200,000-token context and a large output cap can consume far more capacity than dozens of short classification calls. A semaphore that counts requests treats those as equal. Count estimated input tokens plus a reserved output allowance, and keep separate pools for separate routes. That is the bulkhead pattern applied to model capacity.

Load shedding also needs fairness. A single tenant running a batch job should not starve interactive users. Per-tenant concurrency caps, weighted queues, and a reserved interactive pool are all reasonable. The details depend on your product, but the principle is the same: the system should fail selectively, not uniformly.

One warning: shedding is only graceful if the client cooperates. A 429 that triggers an immediate client retry is not shedding; it is a suggestion. Return a clear retry delay, honor it in your own SDK, and cap retries. If the client ignores it, the server's only remaining option is to disconnect, which is a worse experience for everyone.

Once the work is too large for the remaining budget, or the system is under sustained pressure, the right move may be to stop holding the connection open at all. That is where queue handoff comes in.

Queue handoff: stop holding the connection

There is a point where waiting is no longer a product. If the remaining budget is smaller than the realistic completion time, or the provider is unavailable and the breaker is open, a synchronous request cannot succeed gracefully. The honest move is to accept the work, return immediately, and deliver the result later.

Queue handoff changes the contract from "request/response" to "submit/observe." The API returns 202 Accepted with a job identifier, and the client polls a status endpoint, subscribes to server-sent events, or receives a webhook. The UI moves from a spinner to a queued state with a clear explanation and a way to leave the page.

A minimal job record:

type AiJobStatus = "queued" | "running" | "succeeded" | "failed" | "canceled";

type AiJob = {
  id: string;
  idempotencyKey: string;
  tenantId: string;
  userId: string;
  route: string;
  status: AiJobStatus;
  input: unknown;
  result: unknown | null;
  errorClass: string | null;
  deadlineAt: Date;
  attempts: number;
  createdAt: Date;
  updatedAt: Date;
};

The idempotencyKey should be unique in the database. The worker should claim jobs with a visibility timeout, so a crashed worker does not leave a job stuck in running forever. Failed jobs need a bounded retry policy and a dead-letter path. Cancellation should be a state transition, not a delete, so the client can still explain what happened.

The queue is not an excuse to forget the budget. Each job carries its own deadline, and the worker checks it before starting and between steps. A job that has been in the queue for an hour should not suddenly run a twelve-second interactive flow; it should either use a batch-appropriate budget or fail with a clear reason.

Queueing is also a cost decision. A queue that grows without bound during an incident becomes a second outage: workers process stale work, users receive answers to questions they have already abandoned, and the backlog delays recovery. Cap the queue, expire jobs aggressively, and prefer rejecting new low-priority work over storing an unbounded promise.

The UX copy matters here. "This is taking longer than usual. We can keep working and notify you when it is ready" is a legitimate product state. "Please wait" with no escape is not. If you show a position or an estimate, it must be grounded in real queue data. A fake "position 2" is worse than no position at all.

Idempotency is the glue between retries, queues, and side effects

Every degradation strategy above assumes that repeating an operation is safe. Retries repeat provider calls. Hedges duplicate them. Queue workers redeliver jobs after a crash. A user who refreshes the page may resubmit the same form. Without idempotency, each of those mechanisms can create duplicate messages, duplicate charges, or duplicate tool actions.

Idempotency starts at the edge. Generate a key from the stable identity of the user action, not from the HTTP request that happens to carry it. A support reply might use ticket:{ticketId}:draft:{draftVersion}. A checkout assist might use cart:{cartId}:checkout:{attemptId}. A random UUID generated inside the retry loop is not an idempotency key; it is a new operation every time.

At the provider boundary, support varies. Some APIs accept an idempotency key on create operations; others do not expose one for every endpoint. Do not assume that because your HTTP client retries, the provider will deduplicate. Build your own record of the logical operation and its result:

export async function withIdempotency<T>(
  key: string,
  ttlSeconds: number,
  work: () => Promise<T>,
): Promise<T> {
  const existing = await redis.get(`idem:${key}`);
  if (existing) return JSON.parse(existing) as T;

  const locked = await redis.set(`idem:${key}:lock`, "1", {
    NX: true,
    EX: 30,
  });

  if (!locked) {
    throw new Error("operation_in_progress");
  }

  try {
    const result = await work();
    await redis.set(`idem:${key}`, JSON.stringify(result), {
      EX: ttlSeconds,
    });
    return result;
  } finally {
    await redis.del(`idem:${key}:lock`);
  }
}

This sketch has real gaps: a crash between work() and the result write can lose the result, and the lock needs a fencing token or a database transaction to be fully safe. In production, store the idempotency record in the same transactional boundary as the side effect, or use a database unique constraint plus a state machine. Stripe's public documentation is a useful reference for the semantics: a key identifies a request, retries with the same key return the saved response, and reusing a key with different parameters is an error. Stripe's v1 API retains keys for at least 24 hours; your retention window should match the maximum retry and queue lifetime you allow.

Streaming complicates idempotency because the unit of work is not a single response. Key the logical assistant message, not each chunk. If a stream dies and the user asks to continue, create a continuation operation with its own key that references the original message. Do not append a second full answer to the same message record.

Tool calls are the most dangerous case. A model that calls issue_refund twice because a retry duplicated a tool invocation has caused a real-world side effect. Put idempotency keys on the tool boundary too, derived from the conversation, tool name, and normalized arguments. Prefer tools that are naturally idempotent, such as "set status to refunded" rather than "increment refund count." Exactly-once delivery is mostly a fiction; at-least-once delivery plus deduplication is the practical design.

UX states are part of the API contract

A backend that returns a partial answer with a status field is only half the job. The interface has to render the state honestly and give the user a next action. Treat the UI state as a discriminated union, so it is impossible to render "complete" when the data says "partial":

type AiUiState =
  | { kind: "idle" }
  | { kind: "connecting"; startedAt: number }
  | { kind: "streaming"; text: string; startedAt: number }
  | { kind: "slow"; text: string; reason: "first_token" | "inter_chunk" }
  | { kind: "partial"; text: string; missing: string[]; canContinue: boolean }
  | { kind: "stale"; text: string; asOf: string }
  | { kind: "fallback"; text: string; reason: string }
  | { kind: "queued"; jobId: string; canCancel: boolean }
  | { kind: "failed"; message: string; retryable: boolean; requestId?: string };

The state names are boring on purpose. Boring states are easier to test.

A few rules make the difference between a polished degraded experience and an apology:

• Do not show an indefinite spinner. After a threshold, switch to "Taking longer than usual" and explain what is happening. If you know the provider is slow, say that rather than blaming the user's connection.
• Keep the user's input. A timeout should not clear the draft. If the user retries, the original text should still be there.
• Make cancel real. If the user cancels, abort the upstream request where possible and stop billing for work you can stop. If the provider cannot cancel, say the request may still complete.
• Separate "retry" from "resend." Retrying a failed generation should reuse the idempotency key. Sending the same prompt as a new message is a different action and should create a new turn.
• Label degraded output. "Limited answer: live order status is unavailable" is better than a confident paragraph with no status.
• Do not fake progress. A progress bar that moves on a timer is theater. If you do not have real progress, use elapsed time and a clear expectation.
• Respect assistive technology. Announce state changes, not every token. Use a polite live region for "Answer complete" or "Answer paused," and avoid stealing focus while text is streaming.

The visual hierarchy should also distinguish fresh, stale, partial, and fallback content. A small badge is enough if it is consistent. The failure mode to avoid is a degraded answer that looks identical to a normal one; users will make decisions on it without knowing the difference.

Observability: measure the budget, not just the endpoint

If you cannot see which layer consumed the budget, you cannot design the right fallback. Provider dashboards show provider latency. They do not show your queue wait, your retry amplification, your breaker state, or the time your client spent waiting for a tool call. You need a request-level view.

At minimum, record:

Metric	Why it matters
Time to first content token	The user's first signal that something is happening
Inter-chunk gap distribution	Detects stalls that a total-duration metric hides
Total time within budget	The actual product SLO, not raw HTTP latency
Error class by provider/model/route	Separates throttling, overload, auth, and bad input
Retry count and retry ratio	Detects self-inflicted traffic amplification
Hedge count and hedge win rate	Tells you whether duplication is buying anything
Fallback rate by tier	Shows quality drift and hidden outages
Breaker transitions	Distinguishes an open breaker from a slow provider
Cache hit, miss, and stale-serve rate	Shows whether the cache is helping or hiding a problem
Queue depth and age	The leading indicator of a backlog incident
Cost per successful task	Includes retries, hedges, and downgraded attempts

Instrument with a trace per user action. Each attempt should be a child span with the model, provider request ID, status, error code, Retry-After, token counts, and whether it was a hedge or a fallback. When a user reports a bad answer, the trace should let you answer: did retrieval fail, did the first model time out, did the fallback answer, and was the result labeled?

Provider request IDs are worth capturing even when you do not use them every day. OpenAI and Anthropic both return request identifiers, and Anthropic's error responses include a request_id field. When you need to escalate, that ID is the difference between "the API was slow" and a support case with evidence.

Logging needs discipline. Prompts and completions can contain personal data, credentials, and regulated content. Do not log raw prompts by default. Log a hash, a redacted form, or a sampled record with a short retention window and access controls. Debugging value does not justify turning your observability stack into a copy of your users' private data.

Alert on the user-facing symptom. A rise in provider 429s is a useful diagnostic, but the page should fire when the share of requests completing within the interaction budget drops, or when the fallback rate crosses a threshold for a sustained period. Use burn-rate alerts on the budget SLO rather than a single-error alert. One 529 is noise; a 5% fallback rate for ten minutes is a product event.

Finally, test the sad path deliberately. Put a proxy in front of the provider in staging that adds a configurable delay to the first byte, stalls mid-stream, returns 529s, and drops connections. Run the product through each state. Verify that the breaker opens, the UI labels the partial answer, the queue accepts the overflow, and the deterministic fallback renders. A degradation path that has never been exercised is a hypothesis, not a feature.

Putting the pieces together

A coherent request path might look like this:

1. Authenticate the request and derive an idempotency key from the user action.
2. Create a budget from the route's product deadline, not from the provider's default timeout.
3. Check the exact cache. Serve a fresh hit immediately; serve a stale hit only if the route allows it and label it.
4. Ask the admission controller whether this priority class can proceed. If not, return a structured retry or queue response.
5. Check the breaker for the selected model and provider. If it is open, skip to the fallback ladder.
6. Stream the primary model with a first-token deadline, an inter-chunk idle deadline, and the operation deadline.
7. On a retryable failure, retry once if the remaining budget can support it. On a capacity failure, try the next model tier if the route allows it.
8. On a mid-stream failure, persist the partial text and offer continuation. Do not silently restart from zero.
9. If the budget is exhausted or the provider is unavailable, choose in order: labeled stale cache, cheaper evaluated model, deterministic fallback, queue handoff, honest failure.
10. Persist the result and the idempotency record. Emit the trace, the served tier, the degradation reason, and the cost.

The ordering is not sacred. A product where stale data is unacceptable should put the deterministic fallback before the stale cache. A product with a long-running task should queue earlier. The point is that the order is explicit, owned by the product, and testable.

Here is the shape of an orchestrator, with the internal modules left as assumptions:

type AnswerRequest = {
  tenantId: string;
  userId: string;
  route: "support_reply" | "ticket_summary";
  input: string;
  budgetMs?: number;
};

type AnswerResult = {
  text: string;
  mode: "fresh" | "streamed" | "stale" | "fallback" | "queued";
  degraded: boolean;
  requestId?: string;
};

export async function answer(req: AnswerRequest): Promise<AnswerResult> {
  const budget = new Budget(req.budgetMs ?? 12_000);
  const key = `${req.tenantId}:${req.route}:${hash(req.input)}`;

  return withIdempotency(key, 86_400, async () => {
    const cached = await cache.get<CachedAnswer>(key);
    if (cached?.fresh) {
      return { text: cached.text, mode: "fresh", degraded: false };
    }

    if (!admission.allow(req.route)) {
      const job = await queue.enqueue({ ...req, deadlineAt: budget.startedAt + 60_000 });
      return { text: "", mode: "queued", degraded: true, requestId: job.id };
    }

    const choice = chooseModel(req.route, budget, health.snapshot());

    if (breakers.get(choice.model).allow()) {
      try {
        const result = await streamAnswer(req.input, budget, () => {});
        breakers.get(choice.model).recordSuccess();
        await cache.set(key, { text: result.text, createdAt: new Date(), fresh: true });
        return { text: result.text, mode: "streamed", degraded: false };
      } catch {
        breakers.get(choice.model).recordFailure();
      }
    }

    if (cached && cache.mayServeStale(req.route)) {
      return {
        text: cached.text,
        mode: "stale",
        degraded: true,
      };
    }

    const fallback = renderDeterministicFallback(req);
    if (fallback) {
      return { text: fallback, mode: "fallback", degraded: true };
    }

    const job = await queue.enqueue({
      ...req,
      deadlineAt: budget.startedAt + 10 * 60_000,
    });
    return { text: "", mode: "queued", degraded: true, requestId: job.id };
  });
}

This is not a library you can paste into production. It omits authentication, cancellation propagation, streaming to the actual client, tiered fallback, and the persistence transaction for the idempotency record. It is a map of the decisions. The important property is that every branch returns a state the product can render, and every state has a reason.

Comparison table

Strategy	What it buys	Added latency	Added cost	Quality risk	Best used for	Failure mode if misused
Retry with backoff and jitter	Recovers from transient errors	One wait per attempt	Extra requests and tokens	None if the request is idempotent	Network blips, 429/5xx with budget left	Turns a slowdown into a retry storm
Hedged request	Cuts tail latency	Second request starts later	Duplicate work; cancellation may not refund tokens	Two different answers; first is not always best	Short, idempotent, validatable calls	Doubles load during an incident
Streaming	Makes waiting legible; earlier first token	None to total generation	Same tokens, longer connection	Partial output can be mistaken for complete	Chat, drafts, long-form answers	A stalled stream with no watchdog
Partial result	Preserves useful work	Saves the remaining wait	Same or less	Missing facts look like complete facts	Structured extraction, multi-part answers	Downstream treats partial as valid
Stale cache	Instant answer during origin failure	Near zero	Near zero	Out-of-date facts	Slow-moving docs, policies, summaries	Serves stale account or safety data
Cheaper model	Lower latency and cost under pressure	Usually lower	Lower	Quality and format drift	Routes with an evaluated smaller model	Silent downgrade on high-stakes tasks
Deterministic fallback	Predictable answer with no provider	Low	Near zero	Limited coverage	Known intents, status, policy, handoff	Generic error page disguised as help
Circuit breaker	Stops repeated failing calls	Fast rejection	Saves cost	None directly; reduces coverage	Provider/model/route isolation	Global breaker takes down healthy routes
Load shedding	Protects capacity for important work	Fast rejection for shed work	Saves cost	Some users get less	Overload, batch vs interactive contention	Drops critical work without priority rules
Queue handoff	Converts an impossible wait into a job	Immediate response; later result	Storage and worker cost	Result may arrive too late to matter	Long tasks, provider outage, budget exhaustion	Unbounded backlog and stale answers

The table has no universal winner. The right combination is a function of the task's deadline, the cost of a wrong answer, and the cost of asking a human. A code completion and a legal memo should not share a degradation policy.

Design the sad path first

The teams that handle slow APIs well are not the ones with the most sophisticated retry library. They are the ones who decided, before launch, what the product would say at second three, second eight, and second twelve. They wrote the fallback copy. They built the queue. They put a limit on retries that is enforced by the same budget that governs the happy path. They made the breaker visible and the partial state honest.

A practical checklist:

1. Write the timeline for the feature at 1×, 3×, and 10× the expected latency. Name what the user sees at each point.
2. Assign a deadline to every phase and pass it through the call graph. Reserve time for the fallback.
3. Classify every provider error as retryable, downgradeable, queueable, or terminal. Handle the ambiguous 429 explicitly.
4. Pick one layer to own retries. Disable or account for SDK retries elsewhere.
5. Build the deterministic fallback before tuning the prompt. It is the floor that makes aggressive timeouts safe.
6. Make partial, stale, and fallback states first-class in the API and the UI. Label them where the user can see.
7. Add a queue for work that cannot finish inside the interaction budget, with idempotency at the boundary and a bounded lifetime.
8. Instrument time to first token, inter-chunk gaps, retries, fallbacks, breaker transitions, queue age, and cost per successful task.
9. Inject delay and failures in staging. Open the breaker on purpose. Kill the stream mid-answer. Verify the copy and the recovery path.
10. Keep a kill switch for each model route, so a bad deployment or a provider incident can be reduced to a deterministic path in one action.

None of this eliminates slow APIs. Providers will have capacity events, networks will drop streams, and models will occasionally take longer than the product can afford. What changes is the blast radius. A slow dependency becomes a labeled partial answer, a queued job, or a clear next step instead of a blinking cursor and a second click.

The API can be slow. The product does not have to be confusing.

References

• OpenAI, Rate limits and Error codes — retry guidance, Retry-After, slow_down, overload behavior.
• OpenAI, Streaming API responses and Latency optimization — stream events, first-token latency, and why streaming does not reduce total generation time.
• Anthropic, Claude API errors and Streaming messages — 429/529 semantics, long requests, mid-stream errors, and continuation.
• Google, Gemini API rate limits, API errors, and Troubleshooting — quota dimensions and SDK retry behavior.
• AWS, Timeouts, retries, and backoff with jitter and Retry behavior — timeout selection, full jitter, retry quotas.
• Google SRE, Handling Overload — criticality, adaptive client throttling, retry budgets.
• Jeffrey Dean and Luiz André Barroso, The Tail at Scale — hedged and tied requests.
• Envoy, Circuit breaking — network-level limits and retry budgets.
• Stripe, Idempotent requests — key semantics and retention.