Ten thousand requests a day sounds like a scaling problem. It is usually an accounting problem.
Take a support assistant. Each request sends 2,400 tokens of instructions, 6,000 tokens of retrieved help-center content, and an 800-token customer message, then produces a 450-token answer. On a mid-priced model, the tokens cost about $184 a day, or $5,500 a month. Then 2 percent of cases go to a person for five minutes. At a fully loaded $55 an hour, that is 200 reviews a day, 16.7 hours of labor, and roughly $27,500 a month.
The model is now 17 percent of the feature's cost. The rest is the system around it: context that grows, retries that duplicate work, retrieval that stuffs the prompt, cache hits that never happen, tool fees, observability, evaluation, idle capacity, and people.
A rate card tells you what a million tokens cost. It does not tell you what a request costs, or what a successful task costs. The exercise below builds a model of your workload and shows which lever moves the number. Prices are current vendor rates; every workload assumption is stated.
The number that matters is cost per successful task
"Requests per day" is an input, not an outcome. A request can fail, time out, return an unusable answer, get corrected by an agent, or get escalated. Each outcome has a cost, and only some produce value.
Define the unit of work first. For support, it might be "a conversation resolved without human intervention and without a follow-up complaint." For a classifier, "a label that passes audit." For a coding assistant, "a pull request merged without a revert." The definition determines the numerator and the denominator.
cost per successful task =
(model tokens + tool fees + retrieval + infrastructure
+ observability + evaluation + human review + rework)
/ tasks accepted as successful
Most teams compute only the first term and divide by total requests. That number is too low twice: it ignores everything after the model call, and it counts failures as if they produced value. At 10,000 requests a day, a 92 percent success rate means 800 requests a day consumed tokens without producing the outcome. Those requests do not disappear; they raise the cost of every success.
Keep two numbers: the fully loaded cost per success for pricing and staffing, and the marginal cost per request for deciding whether to add another retry or retrieved document. The first request of the day carries cache warming and idle infrastructure; the ten-thousandth mostly carries tokens.
Token accounting: four buckets, not one
Every major provider now bills input in more than one category. The ratios are large. On GPT-6.1 Sol, ordinary input is $2.00 per million tokens, cached input is $0.10, a 30-minute cache write is $2.50, and output is $10.00. Claude Sonnet 5.5 lists the same $2.00 input, $0.10 cache read, $2.50 five-minute cache write, and $10.00 output. Gemini 4 Argon lists $2.00 input, $0.10 cached input, and $10.00 output during its introductory period.
| Bucket | Meaning | Typical multiple of ordinary input |
|---|---|---|
| Ordinary input | Processed fresh on this request | 1.0x |
| Cache write | Written into a reusable prefix | 1.25x short TTL, 2x or more long TTL |
| Cache read | Served from a matching prefix | 0.05x to 0.10x |
| Output | Generated tokens, including reasoning tokens where applicable | 5x to 10x |
Tool usage is a fifth bucket. OpenAI lists web search at $10 per 1,000 calls plus search-content tokens at model rates, file search at $2.50 per 1,000 calls plus $0.10 per GB-day of storage, and hosted containers at $0.03 per GB per 20-minute session. Anthropic lists web search at $10 per 1,000 searches. If 20 percent of 10,000 daily requests trigger one web search, that is 2,000 calls and $20 a day before the tokens those searches add.
The accounting trap is that input_tokens means different things. In OpenAI's Responses API it is the total input, and input_tokens_details.cached_tokens plus cache_write_tokens are subsets; ordinary input is the total minus both. In Anthropic's Messages API, input_tokens is only the uncached input after the last breakpoint; cache reads and writes are separate and must be added to get total input. Code that treats the field as interchangeable will undercount Anthropic or double-count OpenAI.
Tokenization is the second trap: the same text does not produce the same count across model families, and counts can change with a snapshot. Reconcile against the billed usage object rather than an OpenAI tokenizer. Averages are the third: a mean of 9,200 input tokens can hide a p50 of 6,000 and a p95 of 28,000. The p95 hits rate limits and blows up the invoice. Keep histograms by endpoint, model, tenant, and prompt version.
A worked day at 10,000 requests
These assumptions are illustrative, not a benchmark. Edit them.
| Assumption | Value |
|---|---|
| Requests per day | 10,000 |
| Static system prompt, tools, policy | 2,400 tokens, cached |
| Retrieved context | 6,000 tokens, not cached |
| User message | 800 tokens |
| Output | 450 tokens |
| Ordinary input per request | 6,800 tokens |
| Cache writes per day | 48, one per 30-minute window |
ordinary input: 6,800 × 10,000 = 68,000,000 × $2.00/M = $136.00
cached input: 2,400 × 10,000 = 24,000,000 × $0.10/M = $2.40
cache writes: 48 × 2,400 = 115,200 × $2.50/M = $0.29
output: 450 × 10,000 = 4,500,000 × $10.00/M = $45.00
--------
daily total $183.69
That is $5,511 for 30 days, or $0.01837 per request. Output is 4.9 percent of input tokens but 24.5 percent of the cost, because output is priced at five times ordinary input. Retrieved context is 65 percent of input tokens and 65 percent of cost. The static system prompt—the part everyone optimizes first—is 26 percent of input tokens and 1.5 percent of cost after caching.
The same workload across current models:
| Model | Input $/M | Cached $/M | Output $/M | Daily | Per request | 30 days |
|---|---|---|---|---|---|---|
| GPT-6.1 Sol | 2.00 | 0.10 | 10.00 | $183.69 | $0.01837 | $5,511 |
| GPT-6 Astra | 10.00 | 1.00 | 50.00 | $930.44 | $0.09304 | $27,913 |
| GPT-6 Luna | 0.10 | 0.01 | 0.50 | $9.30 | $0.00093 | $279 |
| Claude Sonnet 5.5 | 2.00 | 0.10 | 10.00 | $183.69 | $0.01837 | $5,511 |
| Claude Haiku 5.5 | 0.10 | 0.01 | 0.50 | $9.30 | $0.00093 | $279 |
| Gemini 4 Argon | 2.00 | 0.10 | 10.00 | $183.40 | $0.01834 | $5,502 |
| Gemini 3.8 Flash | 0.75 | 0.075 | 3.75 | $69.70 | $0.00697 | $2,091 |
The Google rows assume implicit caching serves the prefix; Gemini 4 Argon lacks explicit caching. The Flash row includes its $0.50 per million tokens per hour cache-storage charge, about $0.86 a month for a 2,400-token prefix. OpenAI and Anthropic rows include the cache-write premium. Batch, retries, tools, embeddings, and infrastructure are excluded.
Model choice is the largest lever: Astra to Sol cuts the token bill 80 percent, Sol to Luna 95 percent. Whether either is rational depends on success and review cost, which the table does not show. A cheap model that escalates twice as often can cost more per success.
Price timing matters: Gemini 4 Argon's $2/$10 is introductory through November 30, 2026, then $4/$20. GPT-5.6 Sol's promotional pricing runs through at least November 21, 2026. Model post-promotion prices in the base case.
Context growth is a compounding cost
A single request is easy. A conversation is not, because every turn resends history. If the system prompt is S, each user turn adds u, each assistant turn adds a, and the conversation runs T turns:
I(T) = T × (S + u) + (u + a) × T × (T - 1) / 2
The second term is quadratic. With S = 2,000, u = 150, a = 250, and T = 10, total input is 21,500 + 18,000 = 39,500 tokens. Budgeting every turn at the first turn's size gives 21,500 and misses by 84 percent. At 20 turns, the history term alone is 76,000 tokens, and the total is 119,000. The twentieth turn carries about 9,750 input tokens before the model replies.
This is why "keep the whole conversation" is a cost decision, not only a quality decision. Summarization can backfire if it changes the cached prefix: the saving in input tokens gets partly offset by a cache miss. Cap history by tokens, not turns. Keep recent turns verbatim, summarize older turns into a stable state object, and re-retrieve only when the query changes. Retrieved context is resent every turn even when it has not changed, which is usually the biggest waste in support and research assistants.
Set a budget per category before writing the prompt: instructions, evidence, history, output. Enforce it in code. A prompt that can grow without a ceiling is a cost center with a natural-language interface.
Caching: a discount with a hit-rate precondition
Prompt caching is prefix matching, not storage. The provider hashes the rendered prefix up to a breakpoint. If a later request has the same prefix and reaches a machine holding the corresponding state, those tokens are served cheaply. If the prefix changes, the entry expires, or the request routes elsewhere, you pay again.
The mechanics differ enough to need provider-specific assumptions. OpenAI enables caching by default, but GPT-5.6 and later expose explicit breakpoints and lifetimes: a 30-minute write costs 1.25x input, a 12-hour write 2.5x, and reads 0.1x (0.05x on GPT-6.1 Sol). The minimum prefix is 1,024 tokens. Cached state lives on individual machines, and traffic above roughly 15 requests per minute can cause overflow routing, so hit rates can fall as you scale. Anthropic defaults to five minutes, refreshed free on a read: a five-minute write costs 1.25x, a one-hour write 2x, reads 0.1x (0.05x on Sonnet 5.5 and Opus 5.5), and minimums range from 512 to 4,096 tokens. Gemini 4 Argon uses implicit caching only, with cached input 95 percent below input during the introductory period.
Let w be the write multiple, r the read multiple, and h the hit rate. The effective input multiple is h × r + (1 - h) × w. Caching beats no caching below 1, which rearranges to:
h* = (w - 1) / (w - r)
For a 1.25x write and 0.1x read, break-even is 21.7 percent. At 0.05x read, 20.8 percent. For a 2.5x write and 0.1x read, 62.5 percent. For a 2x one-hour write, 52.6 percent. This is the number to put next to the cache dashboard, not the hit rate alone. A 40 percent hit rate is excellent on a 30-minute cache and a loss on a 12-hour cache.
Caching a prefix that changes every request is worse than not caching. If a timestamp sits before the breakpoint, every request is a miss and a write; you pay 1.25x input for the privilege of never getting a read. Put volatile content after the breakpoint. A deploy that changes tool definitions or ordering invalidates the cache for everyone, so prewarm or accept a write spike. Per-tenant cache keys reduce hit rates but prevent cross-tenant probing; decide which risk matters more.
Retries, timeouts, and the requests you pay for twice
A retry adds latency, consumes request quota, and may consume tokens. A 429 or 503 before generation normally does not bill tokens. A timeout after the provider has processed the prompt may be billable, because the work was done even if your client gave up. A retry after malformed structured output is fully billable twice: the first attempt generated tokens, and the second generates them again.
Model billable retries separately from technical retries. If p is the fraction of requests that fail in a way that causes a billable duplicate, expected attempts are 1 / (1 - p). At 3 percent, cost rises 3.1 percent; at 15 percent, 17.6 percent. If failures are not billable, token cost stays flat, but request rate still rises, which can trigger more 429s and more retries.
Retry storms turn a small bill into a large one. An SDK with three automatic retries inside an application loop with three retries can turn one request into nine. Disable one layer. Follow Retry-After, add jitter, cap attempts, and use a deadline for the whole operation; a per-attempt timeout can let a request live for minutes. Put an idempotency key on downstream side effects—tickets, emails, charges—not just the model call.
Fallbacks are not retries. A fallback to a cheaper model changes both cost and quality. A path that succeeds 80 percent of the time at half the price is a blended cost with a new success rate, not a 50 percent saving. Measure it separately or it hides in the average.
Retrieval and embeddings: cheap vectors, expensive context
Embeddings are usually the least expensive part of retrieval, which makes them a distraction from the part that costs money: the retrieved tokens.
Fifty thousand chunks at 400 tokens each is 20 million tokens. OpenAI's text-embedding-3-small costs $0.02 per million, so a full re-embed is $0.40. Query embeddings at 50 tokens per request, 10,000 requests a day, are 500,000 tokens, or $0.01 a day. Re-embedding weekly costs less than $2 a month. Freshness, chunking, and retrieval precision are the real variables.
Storage is small in raw terms. A 1,536-dimension float32 vector is 6,144 bytes. Fifty thousand vectors are about 293 MiB before index overhead. An approximate nearest-neighbor index adds graph links, payloads, and replicas, so measure the on-disk size. A managed vector database may have a monthly floor that matters more at this volume than marginal storage. Idle capacity is a line item even when the data is tiny.
The expensive part is what retrieval does to the prompt. In the worked example, 6,000 retrieved tokens per request cost $120 a day at $2 per million. Every extra 1,000 tokens costs $20 a day, or about $600 a month. Returning eight chunks when three would do is five chunks of noise on every request. Measure retrieval by cost per successful task, not recall alone. A retriever that cuts escalation by one percentage point can pay for a lot of tokens; one that adds 2,000 tokens and changes nothing is pure cost.
The costs that are not on the model invoice
Human review. At 2 percent of 10,000 requests, five minutes each, and $55 an hour, review costs $917 a day, or $27,500 a month—five times the token bill. At 5 percent and eight minutes, it is $3,667 a day, or $110,000 a month. The review rate is a first-class unit-economics variable, not a support detail. Sample aggressively, route by confidence, and measure the time distribution. A review that takes twelve minutes instead of five changes the business case more than a 20 percent token-price change.
Observability. Traces, token accounting, latency, cache-hit rates, and errors are necessary. Storage is often trivial: 10 KB per request at 10,000 requests a day is 100 MB a day, or about 3 GB for 30 days. The real cost is engineering time and high-cardinality queries. Instrument the usage object at the boundary, tag calls with prompt version, model, tenant, and outcome, and sample raw prompts rather than storing them all.
Evaluation. A 500-case eval with 2,000 input and 300 output tokens per case on GPT-6.1 Sol costs about $3.50: 1 million input at $2, plus 150,000 output at $10. Four runs a day is $14, or $420 a month. That is cheap compared with a regression. The expensive version is an LLM judge on every production request; use a small judge for routine checks and a large one for calibration.
Tools, storage, and rework. Web search, file search, and hosted containers are billed separately and can dominate agentic workloads. Raw prompt logs can become the largest data store if retention is unlimited; decide retention by data class and delete what you do not need. A wrong answer that a customer calls about, a label an auditor reverses, or a document an employee rewrites is a cost. Pick a proxy—reopen rate, edit distance, time-to-resolution—and put a dollar value on it, even if approximate.
Rate limits, traffic shape, and idle capacity
Ten thousand requests a day is 6.94 requests a minute on average. That is tiny. The average is almost never what the provider sees. Traffic arrives in morning peaks, campaign spikes, midnight batch jobs, and retry storms.
Anthropic's Start tier for Claude Sonnet 5.5 lists 1,000 requests per minute, 2,000,000 input tokens per minute, and 400,000 output tokens per minute. OpenAI's Build tier lists 5,000 RPM and 1,000,000 TPM for Sol, Terra, and Astra, and 2,000,000 for Luna. At 9,200 input tokens per request, 1,000,000 TPM is about 108 RPM if every request is full-size. A burst of 200 such requests in one minute is 1.84 million tokens, over the Build limit even though the daily average is 6.94 RPM.
Plan for peak-minute tokens, not daily requests. Queue with a concurrency limit, smooth traffic, and handle 429s with Retry-After; more concurrency can make a token-per-minute limit worse. Little's Law:
concurrency = arrival rate × time in system
At 0.116 requests per second and 8 seconds per request, average concurrency is 0.93. At a 10x peak, it is 9.3. If tool calls push time in system to 30 seconds, the same peak needs 35 concurrent requests. Measure time in system including retries and tools, not just time-to-first-token.
Idle capacity is the cost of being ready for the peak. A fixed monthly pool beats pay-as-you-go when:
monthly fixed cost < variable cost per request × monthly requests
If a reserved pool quote is $2,000 a month and variable cost is $0.01837 per request, break-even is about 108,900 requests a month, or 3,630 a day. At 300,000 requests a month, the pool looks cheaper on paper. But a pool is not free capacity for every model, tool, and region, and it can create idle cost if traffic is seasonal. Model utilization by hour, not by month.
Batch, routing, and the order of operations
The cheapest token is the one you do not send. The second cheapest is the one you send asynchronously at a discount.
OpenAI's Batch API is 50 percent cheaper than synchronous pricing and completes within 24 hours. Anthropic's Message Batches API is also 50 percent cheaper, with most batches under an hour and a 24-hour expiry. If 60 percent of workload is offline tagging, summarization, or backfill, blended cost is synchronous cost × (1 - 0.5 × batchable_share), or 70 percent of synchronous cost. The trade-off is latency, operational complexity, and cache behavior. Anthropic says batch cache hits are best-effort, observed from 30 to 98 percent; do not assume the synchronous hit rate carries over.
Route by difficulty. If the small model costs c_s, the large model c_l, and escalation is e, blended cost is c_s + e × c_l. With GPT-6 Luna at $9.30 a day and GPT-6.1 Sol at $183.69, a 25 percent escalation gives $55.22 a day, 70 percent cheaper than always using Sol. The token break-even escalation rate is 1 - c_s / c_l, or 94.9 percent. In practice the small model may need more retries, produce more invalid output, and trigger more review. Measure cost per successful task.
Order matters more than any single trick: remove unnecessary tokens, cache the stable prefix, batch what can wait, route easy work to a smaller model, then negotiate volume pricing. A 10 percent discount on a workload that is 40 percent waste is not a strategy.
Unit economics: per request versus per successful task
Add the non-token costs. The assumptions are illustrative; the structure is the point.
| Monthly line item | Assumption | Cost |
|---|---|---|
| Model tokens, GPT-6.1 Sol | $183.69/day × 30 | $5,511 |
| Query embeddings | 500,000 tokens/day at $0.02/M | $0.30 |
| Vector database and retrieval | Illustrative fixed plus usage | $1,500 |
| Observability and trace storage | Illustrative | $400 |
| Evaluation | 4 runs/day at $3.50 | $420 |
| Human review | 2% × 5 min × $55/hour | $27,500 |
| Total | $35,331 |
That is $0.1178 per request at 300,000 requests a month. At a 92 percent success rate:
$35,331 / (300,000 × 0.92) = $0.1280 per success
The token-only number was $0.0184 per request. The fully loaded cost is seven times higher. At $0.50 per successful task, contribution margin is 74 percent. At $0.10, the feature loses money on every success. Neither conclusion is visible from a price per million tokens.
Sensitivity matters more than precision. A one-percentage-point change in escalation changes monthly review cost by $13,750 at these assumptions. A 10 percent change in token price changes monthly model cost by $551. A 10 percent change in review time changes monthly review cost by $2,750. The human variable dominates. That is not an argument to ignore tokens; it is an argument to instrument both and stop treating the model invoice as the business case.
The quality-adjusted rule is similar: if a cheaper model raises escalation from 2 to 4 percent, the token saving is erased by review. Use the cheapest model whose fully loaded cost per success is lowest, subject to the quality floor.
A decision framework for your own feature
-
Define success, retry, and escalation. If the definition is fuzzy, the denominator will be too.
-
Instrument before optimizing. Log the usage object, model, prompt version, cache fields, latency, outcome, and tenant. Reconcile logged tokens against the invoice for one week.
-
Set a token budget and separate static from dynamic context. Instructions, retrieved context, history, output. Measure p50 and p95, put stable content first, breakpoint after it, and verify cache reads. Remove timestamps, names, and request IDs from the cached prefix.
-
Price the cache honestly. Compute the break-even hit rate for your TTL. If measured hits are below it, shorten the TTL, move the breakpoint, or stop caching that prefix.
-
Cap and compress history. Use a token budget, not a turn count. Summarize older turns into a stable state object.
-
Route and batch. Send easy requests to a smaller model, offline work to batch, hard cases to the expensive model. Measure escalation and success on each path.
-
Add the human and platform lines. Review rate, minutes per review, loaded hourly cost, tool fees, vector database floor, observability, eval compute, storage, and retention. Label every non-official number as an assumption.
-
Set budgets, alarms, and review triggers. Provider spend limits, per-tenant quotas, a daily cost anomaly alert, and a quarterly re-run when prices or models change. A hard limit that returns 429s is better than a surprise invoice, provided the product handles it gracefully. Report cost per successful task, not cost per token.
Instrumenting the call site
The model is only as good as the usage data at the boundary. This Python example uses the Responses API and GPT-6.1 Sol prices, treating input_tokens as the total and subtracting the cache subsets.
import os
from dataclasses import dataclass
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"], max_retries=0)
@dataclass(frozen=True)
class TokenUsage:
ordinary_input: int
cached_input: int
cache_write: int
output: int
def token_cost(usage: TokenUsage) -> float:
return (
usage.ordinary_input * 2.00
+ usage.cached_input * 0.10
+ usage.cache_write * 2.50
+ usage.output * 10.00
) / 1_000_000
def ask(question: str, system: str) -> tuple[str, float]:
response = client.responses.create(
model="gpt-6.1-sol",
input=[
{
"role": "developer",
"content": [
{
"type": "input_text",
"text": system,
"prompt_cache_breakpoint": {"mode": "explicit"},
}
],
},
{"role": "user", "content": question},
],
prompt_cache_options={"mode": "explicit", "ttl": "30m"},
max_output_tokens=600,
)
details = response.usage.input_tokens_details
cached = details.cached_tokens or 0
written = details.cache_write_tokens or 0
usage = TokenUsage(
ordinary_input=response.usage.input_tokens - cached - written,
cached_input=cached,
cache_write=written,
output=response.usage.output_tokens,
)
return response.output_text, token_cost(usage)
Two details matter. max_retries=0 disables hidden SDK retries so the application owns the retry budget. The breakpoint sits on the system text, not the user message, so the question is not written to cache.
The Anthropic version looks similar, but input_tokens is already the uncached portion, so you add the cache fields rather than subtracting them:
cost = (
usage.input_tokens * 2.00
+ (usage.cache_read_input_tokens or 0) * 0.10
+ (usage.cache_creation_input_tokens or 0) * 2.50
+ usage.output_tokens * 10.00
) / 1_000_000
Neither example includes retries, fallbacks, tool fees, or review; attribute those to the same request ID in the orchestration layer. Without a shared key, the cost model fragments into four systems nobody reconciles.
Record cost at the call, not by parsing invoices. Invoices aggregate by model and include credits and committed-use drawdown; the usage object is the only per-request source of truth. Compare the two monthly; a gap of more than a few percent is an attribution bug.
Conclusion: the number to put in the deck
At 10,000 requests a day, the token bill for a mid-priced model on this workload is a few thousand dollars a month, not a few hundred and not a few hundred thousand. The exact number depends on model, context budget, cache hit rate, and output length. The more important number is the fully loaded cost per successful task, usually dominated by something other than tokens: human review, retrieval infrastructure, retries, or a success rate lower than the demo suggested.
The decision framework is therefore not "which model is cheapest per million tokens." It is:
- Measure the token distribution and reconcile it against the invoice.
- Cache only the stable prefix, and check the break-even hit rate.
- Cap context growth; history is quadratic and retrieved context is resent every turn.
- Put a dollar value on human review, retries, and failed tasks.
- Route easy work to smaller models, batch what can wait, and report cost per successful task.
Take the fully loaded cost per successful task and its sensitivity to the escalation rate into the planning meeting. The token price is a vendor decision; the escalation rate, context budget, and review process are yours. Those determine whether the feature is a business or an expensive demo.
The final check is simple. Divide total monthly cost by the number of tasks a customer would accept. If that is below the price the customer pays and above the next-best alternative, you have a feature. If not, prompt tuning will not fix the unit economics; change the workflow, the model, or the scope.