A cache hit feels like free money until someone reads the answer.
Consider a support assistant with a 22,000-token prompt: policy handbook, tone guide, tool definitions, and a dozen worked examples. The first request of the morning processes all of it. The next request, ten seconds later, reuses the prefix and pays a fraction of the input price. Multiply that across a day and the savings look excellent.
Now change the refund policy. The handbook is regenerated, the new version is uploaded, and the prompt is rebuilt with the updated text. The next request should see the new policy. It will, as long as the application actually sends the new text in the same prefix position. But if the cache key, the prompt assembly, or the retrieval layer is holding on to the old bundle, the model will answer from a perfectly cached, perfectly stale prefix. The invoice shows a discount. The customer gets last quarter's rules.
Prompt caching is one of the highest-leverage cost optimizations available for AI features, and one of the easiest to misunderstand. It is not a response cache. It does not decide whether your data is current. It caches the model's computation over an exact token prefix, and it will happily reuse that computation for as long as the prefix matches and the entry lives.
This article covers both sides: the arithmetic that determines whether caching saves money, and the failure modes that determine whether it quietly serves the wrong context.
What is actually being cached
When a provider caches a prompt, it stores intermediate model state — commonly key/value tensors from the attention layers — for a prefix of the request. On a later request, if the beginning of the prompt is byte-for-byte identical and the request is compatible, the provider can skip recomputing that prefix. The model still generates a fresh answer. The cached thing is computation, not content and not the final response.
That distinction explains most of the behavior that surprises people:
- The cache is prefix-based. A change anywhere in the prefix invalidates everything after it.
- The order of the prompt matters. Tools, system instructions, and messages are typically concatenated in a defined order. Moving a stable block after a volatile one destroys reuse.
- The cache does not know that your database row changed. If your prompt builder sends the old text, the cache will reuse the old text.
- The cache does not make the model deterministic. It saves prefill work; sampling still happens.
- The cache is not a semantic cache. A semantically similar prompt with different wording is a different prefix.
Anthropic's documentation puts the scope plainly: prompt caching references the entire prompt — tools, system, and messages, in that order — up to the block marked with cache_control (Anthropic, prompt caching). OpenAI's implementation similarly requires an exact repeated prefix, with caching available on prompts of at least 1,024 tokens and hits occurring in 128-token increments (OpenAI, prompt caching). Gemini's implicit caching follows the same broad model: put large, common content at the beginning and send similar prefixes close together (Google, context caching).
Different providers expose different controls, but the underlying economics are shared.
The four numbers that decide whether caching pays
Caching discussions usually start with the discount. The discount is the least interesting number. Four quantities determine the outcome:
- Prefix size. How many tokens are stable across requests? A 500-token prefix has little room to save. A 20,000-token policy corpus has a lot.
- Hit rate. What fraction of requests actually reuse the prefix? This is where most implementations fail.
- Write and read multipliers. A cache write usually costs more than normal input; a cache read costs much less. Both vary by provider and TTL.
- TTL and request spacing. A five-minute cache that expires between requests turns every request into a write.
The break-even calculation is simple enough to do on a napkin. Let the normal input cost be 1.0 per request. Suppose a cache write costs 1.25× and a cache read costs 0.1×, which matches Anthropic's standard five-minute cache multipliers. If a fraction h of requests hit the cache, the relative cost is:
cost(h) = (1 - h) × 1.25 + h × 0.1
Set that equal to 1.0 and solve:
1.25 - 1.15h = 1
h = 0.2174
So a five-minute cache needs roughly a 22 percent hit rate to beat no caching at all, assuming every miss rewrites the full prefix. For a one-hour cache with a 2× write multiplier, the break-even hit rate is about 53 percent. Those are not impossible targets, but they are not automatic either. A cache that hits 15 percent of the time is a small loss, not a small win.
The formula also shows why caching can increase spend during an incident. If a provider is slow, requests time out, clients retry, and the cache entry expires between attempts, each retry may pay the write premium without ever producing a usable answer. Caching changes the cost shape of failure, not just the cost shape of success.
A worked example makes the scale concrete. Assume a 20,000-token stable prefix, 1,000 requests per day, a $3 per million input token rate, and a five-minute TTL with a 60 percent hit rate. Ignore output tokens and any batch discount.
- Without caching: 1,000 × 20,000 = 20,000,000 input tokens, or $60 per day.
- With caching: 400 misses × 20,000 × 1.25 = 10,000,000 write-equivalent tokens; 600 hits × 20,000 × 0.1 = 1,200,000 read-equivalent tokens. Total 11,200,000 token-equivalents, or $33.60 per day.
At a 90 percent hit rate, the same workload drops to about $12.90 per day. At a 20 percent hit rate, it is roughly $61.20 — slightly worse than not caching. The discount is real; the hit rate decides whether you get it.
💡 Practical note: Run this calculation with your own prefix size and traffic pattern before adding caching complexity. If your prompts are short, bursty, or mostly unique per request, the math may tell you to spend the engineering time elsewhere.
Why stable-first prompt design is the whole game
Most cache misses are self-inflicted. The prompt contains something that changes on every request near the beginning, and everything after it becomes uncacheable.
Common culprits:
- a timestamp or request ID in the system prompt;
- the user's name, plan, or locale interpolated into the first paragraph;
- retrieved documents placed before the static instructions;
- tool definitions serialized from a dictionary whose key order changes between processes;
- a prompt template that includes a random example selection at the top;
- a model-generated summary inserted before the conversation history;
- whitespace or formatting differences introduced by a template engine.
The fix is architectural, not cosmetic. Build prompts in layers:
[tools] stable
[system instructions] stable
[policy corpus] changes on policy release
[user profile] changes per user
[retrieved context] changes per request
[conversation history] append-only
[current message] changes per request
Place the cache breakpoint after the last layer that is stable across the requests you want to share. If the policy corpus is shared by all users and changes weekly, cache through the policy. If the user profile changes per user, put it after the breakpoint. If retrieved context changes every turn, it belongs at the end, where it doesn't invalidate the instructions.
This layering has a second benefit: it makes cache behavior explainable. When the hit rate drops, you can compare the prefix hash at each layer and see which one changed. Without layering, you're reduced to guessing which interpolation broke the match.
OpenAI's guidance adds a subtle point about tools: keep tool definitions, schemas, and ordering stable, and use mechanisms like allowed_tools or tool_choice to change what is callable rather than removing definitions from the prompt (OpenAI, better prompt caching for GPT-6). Removing or reordering tools changes the prefix and invalidates the cache for the rest of the request. If your agent has a large tool catalog turn by turn, that can erase most of the benefit.
Where the cache goes stale
Prompt caching does not invent stale data. It preserves whatever prefix you send. The stale-context failures come from the space between the source of truth and the prompt builder.
1. The content changed, but the prompt didn't
A policy document is updated in the CMS. The application caches the rendered prompt bundle in Redis for an hour. The next request reads the bundle, sends the old policy, and gets a cache hit. Two caches are now compounding: the provider's prompt cache and your application's rendered-prompt cache.
The provider cache is not the problem. The application cache is. If you cache rendered prompts, key them by the content version or hash of every source document, not just by tenant or feature. Better yet, don't cache the rendered string at all; cache the source documents and render on each request. Rendering 20,000 tokens is cheap compared with answering from the wrong policy.
2. The retrieval layer returns a different document with the same prefix
This one is subtle. Suppose your prompt includes a retrieved knowledge-base article, and the cache breakpoint is placed after the article. The article is updated, but the first 500 tokens are unchanged and the document ID is unchanged. The prefix still matches for those 500 tokens, so the provider reuses the cached computation for that portion. That is fine: the cached computation corresponds to the text you actually sent. The model sees the new text after the breakpoint.
The dangerous version is when the application decides the document is "the same" because the ID is the same, and skips re-fetching it. Now the prompt really does contain old text, and caching simply makes the old text cheaper. Prompt caching didn't cause the staleness; it removed the latency signal that might have revealed it.
3. The conversation branched, and the cache followed the wrong path
Conversation trees make caching interesting. If a user edits an earlier message or regenerates an answer, the new branch shares a prefix with the old one up to the edit point. Caching that shared prefix is correct and efficient. The failure is at the application layer: if the prompt builder selects the wrong branch, or if the "current head" pointer is stale, the model receives a cached prefix from a conversation the user thought they had left.
This is not theoretical for chat products with edit-and-regenerate features. Store the message tree explicitly, resolve the active path before building the prompt, and include the path hash in your logs. If the path is wrong, the cache will faithfully accelerate the wrong context.
4. The prompt was updated, but the cache key wasn't
Providers scope caches by organization and model, and some expose routing hints such as prompt_cache_key. Teams sometimes add their own cache-key namespace and forget to include the prompt version. A new prompt version then lands on the old namespace and, if the prefix is identical, reuses the old computation. That's harmless when the text is identical. It becomes harmful when the "new version" is supposed to change behavior through metadata that isn't actually in the prompt — a model-side flag, a tool permission, a policy selector that the application reads from a different place.
The rule: the cache key may include the prompt version, but the prompt itself must contain every input that affects behavior. A cache should never be the mechanism that decides which instructions apply.
5. The cache outlives the data retention policy
A provider may retain prompt cache entries for minutes or hours. If the prefix contains personal data, the cache extends the window in which that data exists in provider infrastructure. That may be acceptable under your agreement, or it may not. OpenAI's data controls documentation notes that prompt caching can store encrypted key/value tensors as application state, with retention depending on the model and organization settings (OpenAI, data controls). Anthropic and Google publish their own retention behavior.
Don't put secrets, access tokens, or unnecessary personal data in a cacheable prefix. If the data must be in the prompt, put it after the breakpoint, and verify the provider's retention terms for cached content. Caching is a data-processing decision, not only a cost decision.
Automatic, explicit, and provider-specific caching
The three major providers expose caching differently. The differences matter when you're designing for portability or debugging a hit rate.
| Provider | Default behavior | Main control | Typical read discount | Notable constraint |
|---|---|---|---|---|
| Anthropic | Opt-in via cache_control |
Automatic top-level marker or explicit breakpoints; 5-minute or 1-hour TTL | 0.1× base input (lower on some newer models) | Exact prefix match; minimum cacheable length; limited breakpoints |
| OpenAI | Enabled by default on supported models | Automatic prefix caching; prompt_cache_key for routing; TTL options on newer models |
Up to 95% off cached input (model-dependent) | 1,024-token minimum; exact prefix; hits in 128-token increments |
| Google Gemini | Implicit caching on 2.5+ models | Implicit by default; explicit cache objects on supported models | About 90% off cached tokens | Explicit caching not supported on newest models; storage billed for explicit caches |
A few details from the current documentation are worth keeping straight.
Anthropic. The default cache lifetime is five minutes and is refreshed at no additional cost when the cached content is read. A one-hour TTL is available at a higher write rate. With standard multipliers, a five-minute cache write costs 1.25× base input and a read costs 0.1×; the one-hour write costs 2× (Anthropic, pricing). The break-even follows directly: one read pays back a five-minute write, two reads pay back a one-hour write. On newer model families the read multiplier is lower, which changes the arithmetic in your favor; check the current table rather than hard-coding a number.
OpenAI. Caching is automatic for supported models and applies to the whole request prefix, including messages, images, tools, and structured-output schemas. For GPT-5.6 and later, the default minimum cache lifetime is 30 minutes with a 1.25× write rate, and a 12-hour option is available at 2.5×; reads are 0.1× on most models (OpenAI, prompt caching). The key operational detail is routing: a cache hit requires the request to land on infrastructure that has the prefix, so prompt_cache_key exists to keep related traffic together.
Gemini. Implicit caching is on by default for Gemini 2.5 and newer, with minimum input sizes that vary by model. Google reports that savings are passed on when a request hits, but does not guarantee a hit. Explicit caching creates a cache object with a TTL and bills for storage, and it gives a stronger guarantee that a referenced cache will be used. As of October 2026, explicit caching is not supported on Gemini 4 Argon and later models, and the Interactions API supports only implicit caching (Google, context caching). If you need explicit cache objects, verify model support before designing around them.
The practical takeaway is not "pick the provider with the biggest discount." It is "pick the caching model that matches your traffic." Implicit caching is nearly free to adopt and easy to waste. Explicit caching gives control and makes you pay for storage. Automatic caching with a routing key works well when many requests share a prefix but arrive on different connections.
TTL is a product decision
The TTL determines the gap your cache can survive. It also determines the write premium you pay. Choosing it is a traffic-shape question, not a purity test.
A five-minute TTL fits interactive sessions where turns arrive seconds apart. The cache stays warm through a conversation, then expires when the user leaves. That is exactly the pattern Anthropic describes: each hit refreshes the timer, and a long enough gap causes the next request to recompute the full input.
A one-hour TTL fits a shared corpus used by many users throughout the day, or an agent that pauses for human approval and resumes later. It costs more to write, so it needs more reads to pay off. In the earlier example, the break-even hit rate moves from about 22 percent to about 53 percent.
OpenAI's 30-minute and 12-hour options follow the same logic. The longer lifetime is for stable instructions or reference material reused across longer gaps; the shorter one is for session context that changes more often. The documentation explicitly suggests mixing lifetimes in one request: put durable material at the start with a longer TTL and session context later with a shorter one (OpenAI, prompt caching).
A useful heuristic: choose the shortest TTL that covers the typical inter-request gap for the prefix, then measure. If the hit rate is high and the cache is expiring between bursts, move up. If the cache is expiring unused, move down. Don't pay for a 12-hour cache to serve a workload that arrives in one burst every morning.
Measuring cache performance without fooling yourself
Provider dashboards show cached tokens. That is not the same as savings.
Track these per feature, per prompt version, and per model:
- Cache read tokens and cache write tokens, not just total input tokens.
- Hit rate, defined as cache read tokens divided by the sum of read and uncached input tokens. Define it once and keep it consistent.
- Effective input cost per request, computed from the provider's rates, not from the undiscounted total.
- Time to first token and total latency, before and after caching.
- Error and retry rate, because retries can turn a cache write into a repeated cost.
- Output quality, because a cache-related prompt change can silently alter behavior.
The most common measurement error is comparing total spend before and after caching without controlling for traffic. If requests grew 40 percent and spend grew 10 percent, caching may have helped a lot. Normalize by request count and by task.
The second error is treating cache reads as proof that the right prefix was used. A cache read tells you the provider matched a prefix; it does not tell you the prefix contained current data. Log a hash of the stable prefix alongside the response. When a stale-content incident occurs, you can determine whether the prompt builder sent an old prefix or whether the model ignored a new one.
A small instrumentation helper makes this concrete:
import hashlib
from dataclasses import dataclass
from datetime import datetime, timezone
@dataclass
class PromptEnvelope:
stable_prefix: str
volatile_suffix: str
prompt_version: str
source_versions: dict[str, str]
def prefix_hash(self) -> str:
return hashlib.sha256(self.stable_prefix.encode("utf-8")).hexdigest()[:16]
def log_record(self, usage: dict, latency_ms: int) -> dict:
return {
"ts": datetime.now(timezone.utc).isoformat(),
"prompt_version": self.prompt_version,
"prefix_hash": self.prefix_hash(),
"source_versions": self.source_versions,
"cache_read_tokens": usage.get("cache_read_input_tokens", 0),
"cache_write_tokens": usage.get("cache_creation_input_tokens", 0),
"uncached_input_tokens": usage.get("input_tokens", 0),
"output_tokens": usage.get("output_tokens", 0),
"latency_ms": latency_ms,
}
The exact usage field names differ by provider. Normalize them at the edge, then store one schema. The important part is that every record carries the prefix hash and the versions of the source documents that went into it. Without those, a cache hit is an opaque event.
A cache-aware prompt assembly pattern
The cleanest implementation separates stable content from volatile content at the data model level, not with string concatenation at the call site.
from dataclasses import dataclass
from typing import Any
@dataclass(frozen=True)
class StableContext:
policy_version: str
policy_text: str
tool_schema: list[dict]
instructions: str
@dataclass(frozen=True)
class RequestContext:
user_id: str
locale: str
retrieved_documents: list[dict]
conversation: list[dict]
question: str
def build_prompt(stable: StableContext, request: RequestContext) -> PromptEnvelope:
# Everything in this block must be deterministic for a given policy version.
prefix = "\n\n".join([
stable.instructions,
render_tools(stable.tool_schema),
f"<policy version=\"{stable.policy_version}\">\n{stable.policy_text}\n</policy>",
])
suffix = "\n\n".join([
f"<user locale=\"{request.locale}\">",
render_documents(request.retrieved_documents),
render_conversation(request.conversation),
f"<question>{request.question}</question>",
])
return PromptEnvelope(
stable_prefix=prefix,
volatile_suffix=suffix,
prompt_version="support-v7",
source_versions={"policy": stable.policy_version},
)
Two properties make this useful. The stable block is a pure function of versioned inputs, so its hash changes exactly when the policy, tools, or instructions change. The volatile block contains everything that varies per request and belongs after the cache breakpoint.
render_tools needs care. If it serializes a Python dict, sort keys and use a stable separator. If it reads from a database, order the tools explicitly. A tool schema that serializes differently between workers will produce different prefixes for identical requests, and the hit rate will look inexplicably low.
For Anthropic, an explicit breakpoint at the end of the stable block looks like this:
message = client.messages.create(
model=MODEL,
max_tokens=800,
system=[
{
"type": "text",
"text": stable_prefix,
"cache_control": {"type": "ephemeral"},
}
],
messages=[
{"role": "user", "content": volatile_suffix},
],
)
This marks the system block as cacheable. The exact placement matters: a breakpoint after the stable block lets the provider reuse tools, instructions, and policy together. Putting the breakpoint after a per-user profile means every user gets a different prefix and the hit rate collapses.
For OpenAI, the equivalent is structural rather than a marker: keep the stable content at the front, keep tool definitions stable, and set a routing key when you have many requests sharing a prefix. On models that support cache options, choose the TTL explicitly rather than accepting a default that doesn't match your traffic.
The cases where caching is the wrong tool
Caching is not free even when the provider doesn't charge for it. It adds a consistency surface, a debugging dimension, and a temptation to keep prompts monolithic.
Skip prompt caching, or use it sparingly, when:
Requests don't share a prefix. A feature that sends a unique 15,000-token document per request has nothing stable to cache. A semantic cache of answers might help, but that is a different system with different correctness risks.
The prompt is small. Below the provider's minimum cacheable length, caching does nothing. Even above it, a 1,500-token prefix saves little and complicates reasoning about cost.
The prefix changes on every request. Per-user instructions, live prices, or a timestamp at the top will keep the hit rate near zero. Fix the prompt layout first.
You need strict per-request freshness with no versioning discipline. If the organization cannot reliably version and invalidate source content, caching will amplify the inconsistency. Fix the content pipeline before turning on the cache.
The cache would contain regulated data you can't retain. Check the provider's cache retention and your own obligations. "It's only a KV tensor" may not satisfy a reviewer who is asking where the data lives.
You are using caching to hide a latency problem. Caching reduces prefill time for repeated prefixes. It does not fix a slow retrieval step, a serial tool call, or a model that spends 30 seconds reasoning. Measure the request breakdown before attributing the win.
🚨 Production warning: A prompt cache can make a broken rollout look healthy. If the new prompt version is deployed to 10 percent of traffic, the other 90 percent may keep hitting a warm cache built from the old version. Aggregate latency and error rates improve while the new version is barely exercised. Compare cache read tokens and prefix hashes by prompt version during a rollout, not just overall.
A decision framework
Use this as a starting point, then replace the thresholds with measurements from your own workload.
Cache when:
- a stable prefix of at least a few thousand tokens is shared across many requests;
- requests arrive within the TTL window, or the prefix is stable enough for a longer TTL;
- the content pipeline has explicit versions, and the prompt is rebuilt from those versions;
- you can log cache read/write tokens, prefix hashes, and prompt versions;
- the provider's retention terms fit the data.
Don't cache when:
- the prefix is mostly unique per request;
- the prompt is below the minimum cacheable size;
- the content changes constantly and has no versioning;
- you cannot observe hit rate or explain a stale answer;
- the only goal is to avoid fixing retrieval or latency.
Treat as a correctness review, not a cost review, when:
- the cached prefix contains policy, pricing, eligibility, or safety instructions;
- the feature takes irreversible actions;
- multiple tenants share a prompt but must not share data;
- a rollout changes instructions without changing the prompt version;
- a cache is being used across a conversation branch or an edited message.
The last category deserves emphasis. Cost optimizations are usually safe to roll back. A stale policy answer that triggers a refund, a compliance statement, or a medical recommendation is not. If the cached prefix can change a decision, put a version marker in the prompt, assert that the version matches the source of truth at request time, and log it with the response.
The rule to remember
Cache the instructions, not the truth.
Static instructions, tool definitions, formatting rules, and a versioned policy corpus are good candidates for a long-lived prefix. The current state of the world — the user's account, the latest document, the result of a tool call, the message they just sent — belongs after the breakpoint, where it can change without invalidating the stable work and where a stale value is visible in the request log rather than hidden behind a cache hit.
The financial case for prompt caching is real. In the worked example, a well-designed prefix cut input cost by roughly 44 percent at a 60 percent hit rate and by roughly 79 percent at 90 percent. But the feature only captures that value if the prefix is stable, the TTL matches the traffic, and the content pipeline can prove which version it sent. Get those three things right and caching becomes a quiet multiplier. Get them wrong and it becomes a very efficient way to serve the wrong answer.