The email arrives on a Tuesday: per-token prices for your primary model are changing. Not by five percent. By enough that the spreadsheet you built six months ago now shows the feature running at negative margin on your cheapest plan.
The engineering work that follows is usually framed as a negotiation problem or a finance problem. It is neither. It is an architecture problem that you either solved in advance or are about to solve under deadline pressure, in a codebase where the model name appears in fourteen files and the token budget is an implicit assumption inside a prompt nobody has opened since launch.
Provider prices move. They move down when a new generation lands and up when a provider reprices a tier, changes caching rules, restructures batch discounts, or introduces a cheaper model that is not a drop-in replacement for the one you are using. If your margin depends on the current numbers, your product has a dependency you have not priced in.
This is about designing AI workflows where a price change is a configuration event with a known blast radius, not an incident.
A price change is a scheduled risk, not a surprise
Teams treat price changes like outages: rare, unpredictable, handled when they happen. That framing produces exactly the wrong preparation. Outages are handled with redundancy and retries. Price changes are handled with substitutability and measurement.
The useful analogy is not incident response. It is capacity planning. You know your infrastructure bill will change. You know your traffic will grow. You build forecasts with ranges, not point estimates, and you revisit them. AI inference belongs in the same bucket, with one addition: the price is set by a vendor you do not control and can change with modest notice.
So the design question is not "how do we get the best price today." It is "what is the maximum price increase we can absorb before the feature becomes uneconomic, and what levers do we have between here and there?"
That question has a number. For a feature where inference is 30% of the cost of serving a customer, a provider price doubling pushes it to 60% and probably still works. For a feature where inference is 85%, the same change is existential. Knowing which one you have determines how much engineering the risk justifies.
Measure cost per successful task, not cost per token
Token pricing is a vendor abstraction. It is convenient for the invoice and misleading as a design metric, because it says nothing about how much work your product does to produce a result a user accepts.
A support-reply feature that costs $0.004 per generation sounds cheap until you account for the fact that 22% of drafts are rejected and regenerated, 8% escalate to a second model, and each request carries 3,100 tokens of retrieval context that is often irrelevant. The real number is cost per accepted reply, and it is usually two to four times the naive per-call estimate.
Build the metric explicitly:
from dataclasses import dataclass
@dataclass
class TaskCost:
task_id: str
tenant_id: str
workflow: str
input_tokens: int
cached_input_tokens: int
output_tokens: int
model: str
price_version: str
attempts: int
outcome: str # accepted | rejected | escalated | failed
latency_ms: int
def cost_usd(self, prices: dict) -> float:
p = prices[self.price_version][self.model]
billable_in = self.input_tokens - self.cached_input_tokens
return (
billable_in * p["input"] / 1_000_000
+ self.cached_input_tokens * p["cached_input"] / 1_000_000
+ self.output_tokens * p["output"] / 1_000_000
)
Two fields carry most of the value. price_version lets you recompute historical cost under a new price sheet without re-running anything, which turns "what would this cost at 2x?" from a guess into a query. attempts and outcome let you compute cost_per_accepted, which is the only number that connects to revenue.
SELECT
workflow,
count(*) FILTER (WHERE outcome = 'accepted') AS accepted,
sum(cost_usd) AS total_cost,
sum(cost_usd) / nullif(
count(*) FILTER (WHERE outcome = 'accepted'), 0
) AS cost_per_accepted
FROM ai_task_costs
WHERE created_at > now() - interval '30 days'
GROUP BY workflow
ORDER BY total_cost DESC;
If you do not have this table, you do not know your exposure, and every discussion about a price change becomes an argument between whoever remembers the last invoice and whoever owns the roadmap.
Find where the tokens actually go
Most teams are surprised by their own token distribution. The generation prompt they think of as "the feature" is rarely the largest line item. Context assembly usually is.
Sort your workflows by total tokens by phase and the pattern is remarkably consistent:
- System instructions. Often 400–1,200 tokens, frequently duplicated verbatim in three prompts that drifted apart.
- Retrieved documents. The largest and most variable. A retrieval step that returns eight chunks at 500 tokens each spends 4,000 tokens before the model has seen the question.
- Conversation history. Grows without bound unless someone capped it, and re-sent on every turn.
- Tool results. Can be enormous if a tool returns a full record when the model needed four fields.
- The actual user message. Usually the smallest input by an order of magnitude.
- Output. Often the smallest cost line, unless the feature generates long documents.
That distribution tells you which lever matters. If retrieval is 70% of input tokens, switching models saves less than fixing the retrieval count. If output is dominant, max_tokens and a stricter output contract matter more than the input price.
A useful exercise: for your top three workflows, compute cost per accepted task with the current configuration, then again with these changes applied one at a time.
| Lever | Typical effect | Effort | Risk |
|---|---|---|---|
| Cap retrieved chunks | 20–50% of input tokens | Low | Recall drops on broad questions |
| Summarize older turns instead of replaying them | 30–70% of conversation cost | Medium | Loses exact phrasing users reference |
| Shrink tool result payloads | 10–40% of input tokens | Low | Model may need a second call |
| Prompt caching for stable prefixes | 20–60% on cached input | Low–Medium | Cache misses still cost full price |
| Route simple tasks to a smaller model | 40–80% on routed traffic | Medium | Quality regression on edge cases |
| Batch non-interactive work | Provider-dependent discount | Low | Latency measured in hours |
| Trim output with a schema | 10–30% of output tokens | Low | Schema must be right |
The exact percentages are context-dependent; treat them as a prompt to measure, not a promise. The point is that a model switch is one lever among several, and often not the best one. Teams reach for it first because it is the most legible change on an invoice, then discover it requires the most re-evaluation.
💡 Practical note: prompt caching changes the shape of your cost curve, not just its height. A cached prefix makes long stable instructions nearly free but makes any per-request variation in that prefix expensive. If you inject a timestamp near the top of your system prompt, you have disabled caching for everything after it — and you will never see an error telling you so.
Make the model a configuration value with a contract
If the model name is a string literal in your business logic, a price change is a code change. If it is a row in a table that encodes capability and cost, a price change is an update.
The minimal version is a routing policy:
workflows:
support_reply:
default: standard
tiers:
standard:
model: "${MODEL_STANDARD}"
max_output_tokens: 700
cost_ceiling_usd_per_task: 0.020
economy:
model: "${MODEL_ECONOMY}"
max_output_tokens: 500
cost_ceiling_usd_per_task: 0.006
escalate_to: premium
escalate_when:
- validator_failed
- user_requested_regeneration
- intent in [billing_dispute, legal_threat]
document_summary:
default: economy
tiers:
economy:
model: "${MODEL_ECONOMY}"
max_output_tokens: 1200
cost_ceiling_usd_per_task: 0.015
Two things make this survive a price change. The model identifier is an environment reference rather than a literal, so repointing to a different model is a deploy of a variable, not a search-and-replace. And the cost ceiling is expressed in money per task, so it does not need to be recalculated when the price sheet changes — it is the constraint, and the router enforces it.
The router itself is small:
def choose_tier(workflow: str, ctx: dict, prices: dict) -> str:
policy = POLICIES[workflow]
for tier in policy["preferred_order"]:
spec = policy["tiers"][tier]
estimate = estimate_cost(ctx, spec, prices)
if estimate <= spec["cost_ceiling_usd_per_task"]:
return tier
return policy["fallback_tier"]
estimate_cost uses an input token count from the actual assembled prompt plus a p90 output length for that workflow. Estimating from the real prompt rather than a constant is the difference between a ceiling that holds and one that is decorative.
🧠 The important part: a cost ceiling is a product decision that should live next to the price of the plan, not buried in an inference client. When someone asks "can we afford richer context on the Pro plan," the answer should be a query, not a meeting.
Build a degradation ladder before you need it
The worst outcome of a price change is not a lower margin. It is a scrambled response: someone flips the model to the cheapest option globally, quality collapses on the workflows that needed the good one, support tickets spike, and the rollback happens two days later after the damage.
A degradation ladder makes the response ordered and reversible. Each rung is a specific, pre-evaluated configuration, and the trigger is a margin threshold rather than a feeling.
| Rung | Change | Quality impact | Reversible in |
|---|---|---|---|
| 0 | Current configuration | None | — |
| 1 | Tighten retrieval to top-4 chunks | Small on focused questions | Config |
| 2 | Enable caching, trim system prompt duplication | None | Deploy |
| 3 | Route classification and extraction to economy tier | Measurable, bounded | Config |
| 4 | Summarize history beyond 6 turns | Moderate on long sessions | Deploy |
| 5 | Restrict premium tier to paid plans only | Visible to free users | Config |
| 6 | Disable the workflow; fall back to templates | Large | Config |
The ladder only works if rungs 1–5 have been evaluated before the emergency. That means running your evaluation set against each configuration and recording the delta, so the decision "we can live with rung 3" is based on evidence rather than optimism. If you have no evaluation set, the ladder degenerates into guesswork at the worst possible moment.
There is also a rung that should not exist: "silently reduce quality and say nothing." Users notice. They notice a summary that misses the point, a classification that flips, a reply that reads flatter. The cost of that discovery is trust, and trust is more expensive to restore than inference.
Multi-provider abstraction: less portable than the slide deck claims
Every discussion of price risk eventually produces the same suggestion: abstract over providers so you can switch when prices move. The suggestion is directionally right and routinely oversold.
What actually transfers between providers is narrow:
- The message array shape, roughly.
- Tool definitions, roughly, with different schema dialects and different strictness guarantees.
- Token counting, approximately, and never exactly.
What does not transfer:
- Prompt behavior. A prompt tuned for one model's instruction-following quirks degrades on another. The system prompt is not portable code; it is a trained interaction.
- Tool-calling reliability. Models differ in how often they emit malformed arguments, how they handle ambiguous tool names, and whether they support parallel calls.
- Structured output guarantees. Some providers enforce a JSON Schema strictly; others treat it as guidance. That difference changes your validation code.
- Caching semantics. Prefix caching rules, minimum cacheable lengths, and TTLs differ enough that a caching strategy designed for one provider may do nothing on another.
- Failure modes. Rate-limit shapes, error taxonomies, and timeout behavior all vary.
A thin adapter over the request/response envelope is worth building. It costs a day and buys you the ability to run a comparison. A full portability layer that promises identical behavior across providers is a multi-month project that produces a lowest-common-denominator tool surface and still does not make the prompts portable.
The pragmatic position: keep the provider call behind one interface, keep prompts as versioned artifacts that can be re-tuned per provider, and maintain an evaluation harness that can score a candidate provider on your own tasks. Portability is the ability to measure and migrate deliberately, not the ability to flip a flag.
What discounts and commitments actually buy
Volume commitments, reserved capacity, and enterprise discounts are real and can materially change unit economics. They are also not architecture, for three reasons.
First, they have terms. A committed spend that assumes current volume becomes a liability if the feature usage drops. Second, they expire, and renewal is a negotiation with a different person at a different company mood. Third, they can mask a design problem: a 30% discount on a workflow that is 4x more expensive than it needs to be buys you time, not a fix.
The disciplined approach is to negotiate from a position where you know your cost per successful task, your sensitivity to price, and your credible alternatives. A vendor conversation that starts with "we can move 40% of this workload to a smaller model next quarter" is a different conversation from one that starts with "we cannot change anything."
That is also why the routing policy matters commercially. Optionality is leverage, and optionality you have never exercised is not real. Running 5% of traffic through a secondary provider on a schedule keeps the integration alive and gives you a real data point for the next negotiation.
Run a price shock drill
The single highest-leverage practice is embarrassingly simple: once a quarter, take your current price sheet, multiply the primary model's input and output prices by two, and recompute your unit economics from the ai_task_costs table.
Then answer four questions in writing:
- Which workflows go negative at 2x?
- What is the cheapest configuration change that restores them, and what does it cost in quality?
- How long would it take to ship that change, including evaluation?
- Who decides to pull which rung, and at what threshold?
If question 3 is answered in weeks rather than days, that is the finding. The drill is not about predicting the next price change; it is about measuring your switching latency while nothing is on fire.
A useful artifact from the drill is a one-page table per workflow:
| Workflow | Current cost/accepted | Cost at 2x | Margin at 2x | Cheapest fix | Fix latency | Quality delta |
|---|
Circulate it to whoever owns pricing and whoever owns the roadmap. The conversation that follows is much better than the one that happens after the invoice arrives.
The pricing details that surprise teams
A price sheet is not a single number, and the line items that matter are rarely the ones in the headline. When a provider changes pricing, the change is usually uneven: input and output move by different amounts, cached input gets a separate rate again, and long-context requests may be billed at a different tier entirely.
Four details are worth modeling explicitly because they change which design wins:
The input-to-output ratio. If your workflow sends 6,000 tokens and generates 300, then input pricing dominates and caching and retrieval trimming are your best levers. If it generates a 2,000-word document from a 200-token brief, output pricing dominates and a stricter output contract matters more than anything on the input side. Teams that assume "output is expensive" optimize the wrong phase in exactly half of the workflows I have seen.
Cached input pricing. Caching is usually cheaper per token but not free, and it typically requires an exact prefix match with a minimum cacheable length. That combination has two consequences. First, a system prompt shorter than the minimum gets no benefit regardless of how stable it is. Second, any variable content at the top of the prompt — a timestamp, a request ID, a user name — silently disables caching for the entire remainder. The failure is invisible: the request succeeds, the bill is just higher.
Reasoning or hidden tokens. Some models bill for internal reasoning tokens that never appear in the visible output, and the count can vary substantially per request even with identical inputs. If you are not recording the provider's reported usage rather than estimating from the text, your cost model will drift. Always persist the usage object the API returns; it is the only authoritative count.
Long-context tiers. Several providers price requests above a context threshold differently from requests below it, sometimes for the whole request rather than the excess. A retrieval change that pushes a workflow from just under the threshold to just over it can double the cost of that call without changing its quality at all. If your context length hovers near a tier boundary, that boundary is a design constraint, not a coincidence.
None of these are reasons to distrust the invoice. They are reasons to compute cost from recorded usage and recorded price versions rather than from a mental model of "tokens in, tokens out."
Where a price change becomes a product change
Engineering tends to model cost as a single blended number per customer. Product and finance model it per plan, and those two views diverge in ways that matter when prices move.
Consider a plan with a flat monthly fee and "unlimited" AI usage. The blended cost per customer is comfortable until the top decile of users arrives. If inference is 30% of revenue at the median and 140% at p95, the plan is not marginally profitable — it is a subsidy paid by the median user to the heaviest one. A 40% price increase turns a tolerable skew into a structural loss, and no amount of routing fixes a plan whose top users cost more than they pay.
The fix is rarely a technical one. It is one of: metering with visible limits, a fair-use threshold with a graceful degradation path, or a price change to the plan. All three require the same input, which is cost attribution per tenant per month with a distribution, not an average.
That is why tenant_id on the cost row is not optional bookkeeping. Without it you can see that inference costs rose; with it you can see that four accounts account for half the increase and one of them is on the cheapest plan.
There is also a subtler product effect. When a price change forces a workflow onto a cheaper tier, the users who notice first are the power users, because they push the workflow hardest. They are also the users whose retention you care about most. A cost-driven degradation that is uniform across the customer base looks fair on a slide and lands asymmetrically in reality. If a rung on the ladder has to be pulled, consider scoping it by plan rather than applying it globally, so the users paying for quality keep it.
The awkward cases
Three situations break the tidy version of this advice.
Latency-bound workflows. A cheaper model that is slower is not cheaper if users abandon. Cost per successful task must include abandonment, and abandonment is driven by latency. A workflow with a two-second budget may have exactly one viable configuration regardless of price.
Quality-floor workflows. Some features have a minimum acceptable accuracy below which they should not ship at all. For those, the ladder stops early. The right response to a price increase is to raise the price of the plan or reduce the scope of the feature, not to degrade a workflow that is useless at lower quality. Being explicit about which workflows are in this category prevents a well-meaning cost optimization from quietly breaking trust.
Long-context workflows. Context length is where price changes hurt most, because cost scales with input and long-context pricing is often tiered. A workflow that sends 80,000 tokens per request has very little room to absorb an increase without changing its retrieval or summarization strategy. These are the workflows to instrument first.
Agentic workflows with variable step counts. A single-shot completion has a cost you can predict from prompt length. An agent has a cost distribution determined by how many tool calls the model chooses to make, and that number is not stable across model versions, prompt edits, or input difficulty. A repricing that looks manageable per call can be severe per task if the workflow averages nine steps. Cap the step count per task, record steps as a first-class field on the cost row, and treat the p95 step count as a budget line. An agent whose cost is bounded only by the model's willingness to stop does not have a cost model at all.
A decision framework you can apply this week
If you are starting from nothing, do these in order. Each one is useful on its own, and each one makes the next cheaper.
- Add the cost row. Record input, cached input, output tokens, model, price version, attempts, outcome, and tenant for every AI task. One table, written from the inference wrapper. Without this, everything else is opinion.
- Compute cost per accepted task for your top five workflows and compare it to the naive per-call estimate. The gap tells you where the waste is.
- Find the dominant token consumer in each workflow. Fix that one thing before considering a model change.
- Move model identifiers into configuration and add a per-task cost ceiling expressed in money.
- Build or find a small evaluation set — even 80 labeled cases per workflow — so you can measure quality deltas when you change a tier.
- Define the ladder with five or six rungs, and evaluate each rung once.
- Run the 2x drill and write down the switching latency.
The order matters because measurement without levers is just anxiety, and levers without measurement are just risk. Together they turn a vendor price change from an existential event into a scheduled configuration review.
Prices will change again. The teams that handle it well are not the ones with the best procurement relationship. They are the ones who can answer, in an afternoon, "what does this cost us now, what would it cost at double, and which switch do we flip first" — with numbers, and without a rewrite.