The real latency budget for an AI assistant inside a web app

The user clicks “Summarize” and sees a spinner for eleven seconds. The model itself generated the answer in three. The rest was authentication, database lookup, retrieval, queueing, tool calls, serialization, and waiting for the browser to paint.

That gap is where an AI feature stops feeling like part of a web application and starts feeling like a slow external service. Teams often tune the model’s generation time while ignoring the rest of the request path. The result is a dashboard that still feels sluggish even after the model got faster.

The real latency budget is end-to-end: from the user’s action to the first useful feedback, and from that action to a result the user can trust and use. This article breaks that budget into measurable stages, explains where streaming helps and where it doesn’t, and shows how to design deadlines, fallbacks, and UI states around the actual task.

Measure the user’s wait, not only model generation

For an interactive AI feature, “latency” can mean several different things:

• Input-to-feedback: how long until the UI acknowledges the action.
• Time to first token (TTFT): how long until generated output begins.
• Time to first useful result: when the user has enough information to act.
• Time to complete response: when the model and all required tools are done.
• Time to committed outcome: when an external action is confirmed and durable.

Those are not interchangeable. A stream can produce a token quickly while taking a long time to finish. A draft can appear complete while an output guardrail is still checking it. A tool call may be the actual work, so the first model text is not useful at all.

Instrument each stage with a shared operation ID:

browser action
  → API gateway / authentication
  → application queue wait
  → database and retrieval
  → moderation / validation
  → model prefill and generation
  → tool calls and retries
  → output validation
  → response serialization and render

If you only record total duration and model duration, you know that the model was not the only cost, but not where the delay occurred. Stage-level spans let you see, for example, that retrieval p95 grew after an index update or that a downstream CRM call dominates completion time.

Budget from the interaction backward

Do not start with “the model usually takes four seconds.” Start with what the user is doing and when the result stops being useful.

A user asking an inline editor to rewrite one sentence expects a short interactive cycle. A user launching a bulk analysis of 2,000 records can tolerate a queue and a progress page. A user changing a payment method needs a confirmed result, not merely a fast draft.

For each feature, define:

1. The maximum synchronous wait before the UI changes state.
2. The target time to first useful feedback.
3. The end-to-end deadline for synchronous completion.
4. What happens when the deadline expires.
5. Whether a result after that deadline is still useful.

These are product decisions, not universal constants. A reasonable internal prototype might choose an acknowledgment under a few hundred milliseconds and a synchronous deadline of several seconds, but those numbers need validation with users and real traffic. They are examples, not a service-level standard.

A deadline should be composed from the actual request path. If the user’s interaction budget is 8 seconds, allocating 8 seconds to the model alone leaves no time for retrieval, validation, or rendering. Reserve some budget for the application around the model.

Decompose latency into a budget you can manage

Use a stage budget as an engineering tool. For an illustrative 7-second interactive deadline:

Stage	Example budget	What to measure
Authentication and request parsing	0.2 s	Gateway and application span
Retrieval and context assembly	0.8 s	Query, rerank, document fetch
Model prefill and first token	2.0 s	TTFT, input size, cache reuse
Tool calls	2.5 s	Per-tool latency and queue wait
Final generation and validation	1.2 s	Output tokens, schema and safety checks
Response render reserve	0.3 s	Serialization and browser work

The values are only an example. The benefit is that each owner can see the slice they control and where the budget is consumed. If one stage regularly exceeds its share, the team has a specific optimization or product trade-off to discuss.

Measure distributions, not averages. A mean of 2 seconds can hide a p95 of 12 seconds. For a web feature, p50 explains the usual feel, p95 captures the frustrating tail, and p99 can reveal operational outliers. Segment by model, prompt version, tenant, input length, cache status, and workflow path. If one tenant sends very long documents, a global average will hide its experience.

1. Acknowledge the action before starting expensive work

The application can make the feature feel responsive before the model returns. Acknowledge the click immediately, show which task is running, and let the user know what input is being processed.

That acknowledgment should not claim progress that the backend cannot observe. “Preparing your summary” is fair if a job has been accepted. “Searching 12 documents” is fair only if the system knows that retrieval stage has started. A fake progress bar that advances by timer teaches users to distrust the interface.

For work that might outlive the browser request, create a task record and return a task ID. The client can poll, subscribe to server-sent events, or reconnect later. The backend then owns task progress and the UI can survive a navigation or connection drop.

For a short synchronous task, disable only the action that would create a duplicate. Do not freeze the entire page. Users may need to inspect the source record or cancel the operation while the assistant works.

2. Reduce prefill work before changing models

The model spends time processing the input before it can produce a first token. Large system prompts, long conversation histories, verbose tool schemas, and excessive retrieved context can dominate TTFT.

Useful reductions include:

• sending only the relevant conversation window;
• summarizing older turns with a verified state summary;
• retrieving fewer, higher-quality passages;
• trimming tool descriptions and unused tool definitions;
• moving stable instructions ahead of volatile context when caching is supported;
• avoiding repeated serialization of the same policy text;
• keeping multimodal payloads to what the task actually needs.

Do not optimize token count by deleting context that grounds the answer. Measure answer quality and latency together. A 30% faster answer that now invents policy is not an optimization.

Prompt caching can reduce repeated-prefix processing when the workload and provider support it, but it will not fix slow retrieval or a long serial tool chain. Track cache reads and misses alongside TTFT so you can tell whether the optimization is active.

3. Parallelize only independent work

If the assistant needs account status, order details, and policy documents, some calls may be independent and can run in parallel. Others depend on earlier results. Parallelize the independent calls, but set per-tool timeouts and a total request deadline.

import asyncio

async def load_context(user_id: str, order_id: str):
    async with asyncio.timeout(1.5):
        account_task = asyncio.create_task(fetch_account(user_id))
        order_task = asyncio.create_task(fetch_order(order_id))
        policy_task = asyncio.create_task(fetch_policy("shipping"))
        account, order, policy = await asyncio.gather(
            account_task, order_task, policy_task
        )
        return account, order, policy

This example assumes the three reads are independent and each client propagates cancellation correctly. If fetch_order depends on tenant information from fetch_account, they cannot be safely parallelized without another trusted source for the tenant ID.

Parallel work can lower wall-clock time while increasing concurrent load. Bound concurrency per tenant and per dependency. Do not launch an unbounded fan-out because an agent generated a large list of search queries.

4. Put a deadline around the whole workflow

Individual request timeouts are not enough. A workflow can spend 3 seconds in retrieval, 10 seconds across two model calls, and another 8 seconds retrying a tool. Each component may technically respect its local timeout while the user waits 21 seconds.

Create an absolute deadline at the operation boundary and pass the remaining budget to every stage. Before starting a retry or tool call, check that there is enough time for the result to be useful. If there is not, stop and transition to a queued or timed-out state.

When the deadline expires, decide whether to:

• return a partial but explicitly incomplete result;
• queue the remaining work and provide a status link;
• cancel the operation;
• return a deterministic fallback;
• mark the outcome unknown and reconcile a possible side effect.

Do not report a generic timeout if a write may have committed. The user’s next action depends on whether the system knows the state.

5. Streaming improves perception, not completion time

Streaming can reduce perceived wait because the user sees the answer as it is generated. It does not make the model finish sooner, shorten tool calls, or reduce the input tokens processed before the first output. Use it when partial text is useful and safe to display.

Streaming is a poor fit when:

• the output must pass a safety or policy check before display;
• the output is structured and incomplete prefixes cannot be consumed safely;
• the model may call tools before producing a user-facing answer;
• the feature needs a single committed result, such as a database update;
• the network or client cannot handle reconnect and resume cleanly.

If the output is streamed, distinguish partial content from a completed answer. On disconnect, preserve what arrived but mark it incomplete. Do not concatenate a retried generation with the old partial text unless the protocol explicitly supports continuation; otherwise the result may repeat, contradict, or omit content.

For accessibility, announce that generation has started and when it completes, but avoid reading every token delta aloud. Provide controls to pause or stop if the task can safely be cancelled.

6. Tool calls often dominate the tail

An agent may issue several sequential tools, and each one adds latency. A fast model cannot compensate for a slow CRM call or a chain of dependent APIs.

Measure per-tool latency and distinguish:

• time waiting for a connection or queue slot;
• server processing time;
• retries and backoff;
• response size and parsing;
• time spent waiting for user approval.

Set timeouts based on the operation. A read-only lookup may have a short deadline and a fallback search; a payment operation may need a longer confirmation path and reconciliation rather than a blind timeout. Use circuit breakers when a dependency is persistently failing, and avoid retrying a tool call if it may already have committed a side effect.

If the agent calls the same tool repeatedly, it may be a loop rather than a slow dependency. Detect repeated normalized calls and return a controlled failure before the task consumes its entire budget.

7. Make fallback behavior part of the latency design

When the primary model is slow, fallback choices include waiting, switching to a different model, returning a cached answer, using deterministic logic, or queuing the request. Each changes quality, cost, data routing, and user expectations.

Fallback to a smaller model only if it meets the task contract. A cached answer may be stale. A deterministic fallback may be less polished but more reliable for a known lookup. A queue may preserve quality while changing the interaction from synchronous to asynchronous.

Do not silently change the model or region if the user or tenant has a policy constraint. Record the actual served model and the reason for fallback. If the fallback can change the meaning of the response, label the result or route it for review.

8. Do not make the browser wait for background work

Long-running research, bulk processing, document ingestion, and multi-agent tasks often do not belong in a request-response HTTP cycle. Create a durable job, return its identifier, and let the user continue working.

A queued interaction needs its own contract:

• a stable status page or task reference;
• visible states such as queued, running, waiting for approval, completed, failed;
• a notification when the task finishes, if appropriate;
• a cancellation path with honest semantics;
• a retention and retry policy;
• a way to resume after deploys or worker restarts.

Queues also protect providers and your own worker pools by smoothing bursts. But they can hide backlog growth. Show queue age and estimated wait only when you have a defensible estimate; otherwise show status and a notification option without fabricating a countdown.

9. Test latency under realistic conditions

Measure with production-like input sizes, cold and warm caches, normal concurrency, and realistic tool behavior. A laptop demo with a short prompt and one warm connection does not model the user’s first request after a deploy.

Test scenarios such as:

• a short question with no retrieval;
• a long conversation with compaction;
• a large document and slower ingestion;
• a tool that returns slowly but eventually succeeds;
• 429 and 503 responses with retry guidance;
• a stalled stream and client disconnect;
• a dependency outage during a multi-step workflow;
• concurrent requests from one tenant and many tenants.

OpenAI’s current error documentation distinguishes transient rate limits and model overload from exhausted credits and spend limits. The first class may be retried with backoff; the latter requires an account or budget change. Test the UI and workflow behavior for both rather than treating every 429 as “try again.” (OpenAI, error codes)

Record p50, p95, and p99 by stage and task type. A single average can hide a poor tail, and an end-to-end average can hide which dependency owns it. After every model, prompt, retrieval, or tool change, compare both latency and task quality.

10. Design the interface around the deadline

The UI should communicate what the system is waiting for without exposing meaningless internals. “Searching your documents” is useful if retrieval is actually running. “Waiting for approval” is useful if a human owns the next step. “Model inference 3/6” is usually not.

For each latency state, decide:

• what the user can still do in the rest of the application;
• whether their input is saved;
• whether they can cancel without a side effect;
• whether the operation continues if they navigate away;
• whether a partial result can be used;
• how they will learn about completion or failure.

Make cancellation semantics explicit. Aborting the browser request may stop the client from waiting but not necessarily stop the provider or a downstream tool. If cancellation cannot be guaranteed, say “Stop waiting” or “Cancel pending steps” only when that is what the backend actually does.

Budget the tail, not only the typical request

Users notice the slow tail more than the average. A workflow with three parallel reads waits for the slowest read, so each dependency’s tail contributes to the overall tail. A workflow with several sequential model and tool calls compounds delays directly. If each stage has an occasional slow response, the end-to-end p95 can be much worse than the p95 of any one stage.

For parallel calls, set a common deadline and decide what can be omitted. If policy retrieval finishes but optional account history is slow, the assistant may still provide a limited answer with the missing source disclosed. If authorization lookup is slow, the system should not proceed with a write based on stale permissions.

For sequential calls, remove unnecessary turns. A model call that chooses a tool, another call that paraphrases the tool result, and a third call that formats it may be avoidable if the application can render a known result template. Each extra step adds latency, cost, and another failure point.

Avoid making every latency tail a retry. Retries can pile onto an already slow dependency and create a retry storm. Use a retry budget and respect provider guidance. If the deadline is nearly exhausted, a clear partial result or asynchronous handoff may be better than another attempt that cannot finish in time.

An illustrative end-to-end budget

Suppose an interactive assistant has a 7-second synchronous deadline. A team might reserve 0.3 seconds for request handling and rendering, 0.8 seconds for retrieval, 2 seconds for model prefill and first token, 2.5 seconds for tool calls, and 1.4 seconds for final generation and validation. The exact allocation depends on the feature; the purpose is to make trade-offs explicit.

If retrieval consumes 1.6 seconds at p95, the team has options: improve the index, reduce the number of candidate documents, parallelize independent lookups, or shorten the model’s context. If tool calls consume 4 seconds, moving the model to a faster tier may barely change the experience. If the final validation step needs a second model call, consider whether a deterministic validator can enforce the property instead.

The budget should also account for input size. A short request and a 100-page document may have different service classes and deadlines. Either reject inputs that exceed the synchronous path, route them to a background job, or offer a progress state. Do not let one huge request consume the same worker pool that serves short interactive answers.

When the deadline is exceeded, record which stage consumed the budget and whether side effects had started. The UI should not report only “timed out”; the operational record should identify whether the delay was queue wait, retrieval, provider time, or downstream execution.

Capacity and fairness are part of user-perceived latency

Even a fast model can feel slow when one tenant consumes the shared quota or when batch work competes with interactive traffic. Partition capacity by task class and tenant where the platform allows it. Use queue priorities or separate worker pools so an overnight embedding job does not starve an interactive support assistant.

Apply admission control before accepting more work than the system can complete within its promise. If a request will wait longer than the synchronous deadline, return a queued state immediately rather than holding an HTTP connection and eventually timing out. Make the queue visible to operators through age, depth, and completion rate.

Fairness also matters. A noisy tenant should not consume the full shared budget and degrade everyone else. Use per-tenant request and token budgets, and communicate when a tenant’s quota is exhausted. A quota error should not be presented as a transient provider outage if the remedy is an administrator changing the plan or limit.

If the feature uses caching, report both warm and cold latency. A warm cache may make the demo look excellent while the first request after a deploy or quiet period remains slow. The product should be designed to meet a reasonable experience even when the cache is cold, or the application should prewarm it through a documented process.

Revisit the budget when prompts, retrieval sources, tool schemas, model versions, or traffic patterns change. A prompt that adds a long policy section can increase prefill time even if the model and infrastructure are unchanged. A new approval step can move the slowest part of the task from inference to human wait. Treat the budget as a living product contract and attach it to the feature’s release review.

When a threshold is missed, identify the owner of the slow stage before changing the user experience. If the model is fast but the internal API is slow, switching model providers is unlikely to help. If the user is waiting on a human review, the remedy is queue design and staffing, not a spinner animation. Document the chosen fallback and its owner.

A practical latency review

Before shipping an assistant inside a web app, answer:

1. What is the end-to-end deadline for this user task?
2. What are p50, p95, and p99 for retrieval, model, tools, validation, and render?
3. When does the user get the first useful feedback?
4. Which work is synchronous, and which should be queued?
5. Are retries bounded by the operation deadline and cost budget?
6. Can the browser disconnect without duplicating a write?
7. Is partial output clearly marked and safe to display?
8. What fallback preserves the task’s quality and data policy?
9. Does the interface stay usable while the AI panel is waiting?
10. Can the team identify which stage consumed the time from a trace?

An AI assistant does not need to be instantaneous. It needs to be predictable about what happens while it works and honest when it cannot finish within the interaction. Measure the full path, allocate the budget by stage, and move long work to a durable task when a synchronous answer is no longer useful. That is how latency becomes a product decision rather than a spinner design problem.