Why AI demos look fast and production AI feels slow

The demo answered in nine seconds and the room nodded. In production, the same feature feels broken at four.

Nothing about the model changed between those two moments. What changed is everything around it: the length of the prompt, the number of retrieval calls, the temperature of the cache, the number of people using it at once, and the fact that in the demo someone was talking while the tokens arrived. The stopwatch said nine seconds in both cases. The experience did not.

Latency in AI features is less mysterious than it feels, but it is more distributed than most teams expect. The model is one span in a waterfall that starts at the click and ends when the user believes the work is done. Demos compress that waterfall. Production exposes it.

The demo is a different workload wearing the same UI

A demo is a controlled experiment with the variables chosen to flatter the system. It is not dishonest; it is just not production.

Condition	Demo	Production
Prompt length	Short, hand-written, no history	System prompt, tools, retrieved context, conversation
Retrieval	Pre-loaded or skipped	One to three network calls, sometimes slow
Tools	None, or a single mocked call	Multiple sequential calls with real latency
Cache	Warm from rehearsal	Cold, or invalidated by the last deploy
Concurrency	One user, one request	Hundreds, with queueing at the provider and at your edge
Input quality	Curated	Messy, long, ambiguous
Output length	Short	Whatever the task needs
Failure handling	None	Retries, fallbacks, validation, all on the clock
User attention	Presenter narrating	Someone waiting, often on a phone

Every row adds time or variance. A demo that runs in nine seconds can become a production p50 of 14 seconds and a p95 of 40 without any single component being "slow". That is the ordinary arithmetic of distributed systems, applied to a workload that people experience as a conversation.

The same nine seconds, decomposed twice

It helps to see where a demo's time actually goes. In a rehearsal, the presenter often starts the clock when they click, but the interesting parts happened earlier.

Segment	Demo	Production
Page and session setup	Already loaded	300–900 ms
Auth, tenant, quota	Bypassed or cached	40–120 ms
Query embedding	Precomputed	60–150 ms
Retrieval	One fast query	150–600 ms
Prompt assembly	Short template	20–80 ms
Provider queue	Empty	0–2,000 ms
Prefill	Under 1 s of tokens	0.4–3 s depending on length and cache
Decode	120 tokens	400–900 tokens
Post-processing	None	100–800 ms

The demo's nine seconds were mostly decode, because everything else was removed. Production's nine seconds are a sum of small, individually reasonable costs, plus a queue that can dominate on a bad minute. When a team says "the model got slower", they usually mean one of the middle rows changed and nobody was measuring it separately.

This is also why benchmark charts are a poor guide to product latency. A published tokens-per-second figure describes decode throughput under controlled conditions. It says nothing about your retrieval, your prompt length, your tool calls, or your queue. Use it as an input to a budget, never as a prediction of the user experience.

Latency is a chain, and the model is rarely the whole chain

Draw the waterfall before you optimize anything. For a retrieval-backed assistant, a typical request looks like this:

0 ms      request arrives, auth and tenant lookup
40 ms     rate limit check, quota check
80 ms     query embedding
120 ms    vector search + metadata filter
260 ms    rerank
340 ms    prompt assembly and tokenization
380 ms    provider connection, queue, prefill begins
1,450 ms  first token (TTFT)
1,450 ms  ... decode at ~55 tokens/sec ...
8,900 ms  final token
9,050 ms  output validation, citation check
9,120 ms  persistence, response complete

In that example, the model accounts for 8.5 of the 9.1 seconds, but the model's own prefill is only about a second. The rest is decoding 400 tokens. Change the output to 900 tokens and the total roughly doubles. Change the retrieval to a slower reranker and the first token arrives 300 ms later, which users notice far more than they notice the tail.

The practical consequence: time to first token and total time are different products. A chat interface can feel responsive with a 1.5-second TTFT and a 15-second total if the text arrives steadily. A batch action with no streaming feels dead at 3 seconds even if it completes in 6.

Instrument the chain, not just the endpoint. A single duration_ms on the outer request is enough to alert, but not enough to explain.

Prefill and decode scale differently

Two phases dominate model latency, and they respond to different fixes.

Prefill processes the input. It is compute-heavy and parallel, so it scales with prompt length but not linearly with output. Caching helps here: a repeated prefix can skip most of the work, which is why prompt caching reduces TTFT most on long, stable contexts. On OpenAI's API, caching is enabled by default for supported models, and the documented benefit is cheaper and faster input processing; the discount reaches up to 90 percent of input cost on some models.

Decode generates output one token at a time. It scales with the number of output tokens and depends on the model's throughput on the hardware serving your request. Typical hosted frontier models produce tens of tokens per second; a 500-token answer therefore takes several seconds no matter how fast your retrieval is.

This is why "make the prompt shorter" and "make the answer shorter" are not interchangeable advice. Shortening the prompt reduces TTFT and input cost. Shortening the output reduces total time and output cost, and is usually the larger lever for perceived speed because users wait for the last token.

A 600-token answer at 60 tokens per second costs ten seconds of decode. Asking the model for a 200-token answer with a structured summary cuts that to about three seconds. Add a "show more" affordance for the rare user who wants the long version, and you have changed the experience more than any infrastructure work would.

Streaming changes perception, not duration

Streaming is the most effective perceived-latency technique available, and it is also the most oversold.

It does not make the task finish sooner. The final token still arrives at the same time. What it changes is the moment the user sees progress, which affects abandonment, perceived speed, and the willingness to wait. That is a real product benefit. It is not a latency reduction, and teams that report streaming as a speed improvement are measuring satisfaction while claiming throughput.

Streaming has costs that demos hide:

More failure surface. The connection can drop mid-stream, and partial output is harder to retry cleanly than a single response.
Validation after the fact. You cannot validate a complete JSON object until the stream ends, so structured outputs often can't stream usefully without a partial parser.
Rendering work. Markdown parsing on every token can jank a low-end phone. Batch UI updates to animation frames.
Perceived inconsistency. If the first tokens are wrong and then corrected by the model mid-answer, the user sees the confusion in real time. Non-streamed output hides it, for better and worse.

Use streaming when the user is reading along and can act on partial content. Don't use it when the output must be validated, approved, or applied atomically.

The agent multiplier

Single-call demos are the best case. Production features often become agents, and agents multiply latency in ways that are easy to miss in a design review.

A five-step agent where each step is a model call plus a tool call looks like this:

Step	Model time	Tool time	Cumulative
Plan	1.2 s	—	1.2 s
Search	0.8 s	1.1 s	3.1 s
Fetch record	0.9 s	0.6 s	4.6 s
Draft	2.4 s	—	7.0 s
Validate	0.3 s	0.2 s	7.5 s

Seven and a half seconds for a task that "takes one model call" in the architecture diagram. If the agent retries the search because the first result was empty, add three seconds. If it decides to check a second source, add four more.

Three techniques keep this bounded. Parallelize independent tool calls instead of letting the model serialize them. Cap the number of steps and the wall-clock budget, and fail into a deterministic path when the budget is exhausted. And precompute what you can: if the agent always fetches the same account record, fetch it before the model asks.

The worst pattern is an agent that re-sends the entire transcript on every step. Input tokens grow with each turn, so step five pays for steps one through four again. Caching helps if the prefix is stable, but a conversation that appends tool results between steps often invalidates the prefix precisely where the growth is. Measure tokens per task, not tokens per call.

Queueing is where p50 and p95 divorce

At low concurrency, latency is the sum of service times. At higher concurrency, it is the sum plus the wait.

Little's law gives the intuition: the average number of requests in the system equals arrival rate times average time in the system. If 20 requests arrive per second and each takes 2 seconds, 40 are in flight. If your provider allows 20 concurrent requests, the other 20 wait. The wait is not a model problem; it is a capacity problem, and it shows up as a latency cliff that no amount of prompt engineering fixes.

Three queueing effects explain most "it was fast yesterday" reports:

Provider-side queueing. Your requests wait behind other tenants. This is invisible in your traces except as elevated TTFT with normal prefill token counts.

Rate limits and 429s. A retry after a 429 adds the backoff delay plus the original wait. Aggressive retries under load make the queue longer, which is the classic retry storm in a new costume.

Your own concurrency limits. Connection pools, worker counts, and per-tenant quotas all impose a ceiling. A pool of 10 connections serving 40 concurrent streams will produce waits that look like model slowness.

Measure queue wait explicitly: timestamp when the request enters your system, when it is dispatched to the provider, and when the provider starts responding. The gap between the first two is yours. The gap between the second and TTFT is mostly theirs.

Timeouts deserve their own line in the budget, because a timeout that is too generous is a latency bug. If the client waits 60 seconds for a first token before retrying, the user has already left. Set the timeout from the p99 of successful requests, not from a round number, and decide explicitly what the user sees when it fires. A fast, honest failure with a retry affordance is better than a slow, silent one.

Retries also interact badly with streaming. Once the first token has been rendered, a retry means either appending a second answer to the first or clearing the screen and starting over. Neither is pleasant. Prefer a retry that happens before any output is shown, and treat mid-stream failures as a separate product state with its own copy, not as a generic error toast.

A small capacity model makes the cliff visible before it happens. Suppose each request holds a provider slot for the full duration of the response, the average response takes eight seconds, and your tier allows 60 concurrent requests. The maximum sustainable arrival rate is:

throughput_max = concurrency / average_duration
               = 60 / 8 s
               = 7.5 requests per second

At 7 requests per second you are near the ceiling and queueing is visible at p95. At 8, the queue grows without bound until something times out. The numbers are illustrative, but the shape is universal: capacity is a function of duration, so making responses longer reduces how many users you can serve, even if token throughput is unchanged.

There is a second-order effect that surprises teams. Longer outputs increase duration, which increases the number of in-flight requests, which increases queue wait, which increases the chance the user retries, which increases the arrival rate. Under load, latency and traffic become coupled. A feature that is stable at 5 requests per second can enter a feedback loop at 7 that looks like a provider outage but is really a capacity design problem.

Cold caches and warm demos

Caching is a latency feature with a hit-rate precondition. A demo that runs the same prompt twenty times has a warm cache and a warm connection. Production has neither.

Cache entries expire. OpenAI's default is a short in-memory retention window, with extended retention available on some models; Anthropic's default is five minutes, with a one-hour option at a higher write cost. A traffic pattern with bursts separated by quiet periods will miss more often than the average suggests. A deploy that changes the system prompt or tool definitions invalidates the prefix for everyone at once, producing a synchronized slow period right after release.

Cold starts matter for the parts you run yourself: the embedding service, the reranker, the vector database connection pool, the function that assembles prompts. A serverless function that initializes a client on every invocation adds hundreds of milliseconds that never appear in the demo.

The fix is not "always keep everything warm". It is to know your cache hit rate by route and by hour, prewarm after deploys, and treat a cache miss as a latency event with a budget rather than an accounting detail.

The interface is part of the latency

Users don't experience your model. They experience a sequence of interface states, and some of those states are self-inflicted.

A blocking spinner with no progress indication feels longer than the same wait with a skeleton and a status line. A disabled submit button that re-enables after 12 seconds invites a second click, which produces a duplicate request, which produces a second wait. A page that fetches the user's profile, then the conversation, then the model response in series adds two round trips before the work even starts.

Three interface changes usually beat a 20 percent model speedup:

Render the shell immediately. Show the question, the context, and a placeholder answer region. Perceived wait starts when the user sees their request acknowledged.
Stream or stage. Show retrieval results or a plan before the final answer when the workflow allows it. Even a "checking three sources" status reduces abandonment.
Make completion explicit. A spinner that stops at the last token but leaves the page in a loading state for another two seconds (persistence, analytics, a follow-up fetch) reads as slowness. Sequence the writes so the visible result lands first.

These are not tricks to hide a slow backend. They are honest representations of a multi-stage process. The dishonest version is a fake progress bar that reaches 90 percent and stops, which trains users to distrust every future progress indicator.

Geography and the network you forgot

A request from Sydney to a US-east model endpoint carries a round-trip time that no model optimization can remove. TLS handshakes, DNS, and mobile radio wake-up add more. On a good wired connection this is tens of milliseconds; on a congested mobile network it can be hundreds, and it applies to every tool call and every retrieval hop.

This is one reason regional endpoints exist, and one reason they cost more. OpenAI charges a 10 percent uplift for regional processing on eligible models released on or after March 5, 2026. Anthropic applies a 1.1x multiplier for US-only inference via inference_geo on Claude 4.6 and later. The premium buys latency and residency, not tokens.

If your users are global and your latency budget is tight, the architecture question is not "which model is fastest" but "where does the request terminate, and how many hops does it take before it reaches the model?" A regional edge that handles auth and retrieval, plus a regional model endpoint, is often worth more than a faster model in another hemisphere.

A latency budget per interaction type

"Fast" is meaningless without a target. Set budgets by what the user is doing, then allocate across the chain.

Interaction	TTFT target	Total target	Notes
Inline autocomplete	< 300 ms	< 800 ms	Below 300 ms feels instant; above 1 s users stop waiting
Chat first response	< 1.5 s	< 8 s	Streaming required; show a status if retrieval is slow
Draft generation	< 2 s	< 15 s	User expects to review, not wait idle
Agent task	< 3 s to first status	< 30 s	Progress must be visible; cap steps
Background enrichment	n/a	minutes	Optimize cost, not latency

Allocate the budget explicitly: 200 ms network, 300 ms auth and retrieval, 800 ms prefill, 6 s decode, 500 ms validation and persistence. When a component exceeds its allocation, you know what to fix. Without allocations, every slowdown becomes a debate about the model.

What to measure, and where

The minimum instrumentation for an AI request:

queue_wait_ms — time between arrival and dispatch
ttft_ms — dispatch to first token
decode_ms — first token to last token
output_tokens and tokens_per_second
tool_time_ms per tool call, with the tool name
retrieval_time_ms and result count
validation_time_ms
persist_time_ms
status and retry_count
cache_hit_tokens and cache_write_tokens

Report p50, p90, and p99 by route, model, and tenant. A p50 that looks fine with a p99 above 30 seconds is a feature that a meaningful minority of users experiences as broken.

A synthetic canary is worth more than most dashboards. Run a fixed prompt every minute from the regions your users actually use, and alert on TTFT and total time. Provider incidents often appear in the canary before they appear in your own traffic, because your traffic is a biased sample of the routes you happen to serve.

Techniques, ordered by return on effort

Most teams reach for model speed first because it is the most visible lever. It is rarely the best one.

High return, low effort

Cap output length with a schema or an explicit token budget.
Parallelize retrieval, auth, and profile lookups.
Prewarm caches after deploys.
Stream when the user is reading.
Show a staged status instead of a blank spinner.

High return, medium effort

Add prompt caching for stable prefixes and measure hit rate by route.
Route simple subtasks to a faster model instead of using one model for everything.
Precompute context that the agent always needs.
Cap agent steps and add a wall-clock budget with a deterministic fallback.

Medium return, high effort

Regional endpoints and edge termination.
Hedged requests to reduce tail latency (with care: they double load on the slow path).
Custom inference or provisioned throughput for steady high volume.
Speculative decoding or distilled models for narrow tasks.

Notice that "switch to a faster model" appears in the middle, not at the top. It helps, but it rarely fixes a chain that is 70 percent tool calls and queueing.

When slow is the correct design

There is a version of this problem where the fix is not speed at all.

A reasoning-heavy task that takes 40 seconds and produces a correct answer may be better than a 4-second answer that is wrong 30 percent of the time, because the cost of the wrong answer includes a human review and a customer conversation. A batch enrichment job that takes an hour is fine if nobody is waiting. A compliance check that must complete before a payment is released should be slow and explicit rather than fast and optimistic.

The honest framing is a trade-off between latency, cost, and quality, and the right point depends on the consequence of being wrong. For an autocomplete, latency dominates. For a contract analysis, quality dominates and the interface should communicate that work is happening.

A decision framework for the next slow feature

When someone says the AI feature "feels slow", work through this in order.

Define the experience being judged. First token, complete answer, or task completion? Measure that, not the model call.
Draw the waterfall with real numbers. Include auth, retrieval, tools, prefill, decode, validation, persistence, and rendering.
Separate queue from service. If queue wait is a large share, fix capacity before touching prompts.
Check the cache. Hit rate by route, cold start after deploy, TTL relative to traffic pattern.
Count the output tokens. This is the most common and most fixable cause of a long total time.
Audit the interface. What does the user see at 0, 1, 3, and 10 seconds?
Set a budget and allocate it. Then optimize the component that exceeds its share.
Re-measure with a canary and real users. Lab timings and production timings diverge; assume the divergence is structural until proven otherwise.

The demo was fast because it was small, warm, singular, and narrated. Production is large, cold, concurrent, and silent. Close that gap with measurement and interface design, and the model's actual speed becomes the least interesting part of the conversation.