A team I worked with cut its inference bill by 41 percent in a single sprint. They moved a document-extraction workload from a mid-tier model to a much cheaper one, trimmed the system prompt, and turned on caching. The finance dashboard showed the win immediately.
Three weeks later the unit economics were worse than before. The cheaper model returned valid JSON more often than expected but put values in the wrong fields more often than the old one. Downstream validation caught some of it. Some it did not, and those documents were corrected by an operations team at roughly eleven minutes each. Add the retries, the second-pass prompts, and a small increase in escalations, and the cost per accepted document had risen by about 18 percent.
Token spend is an input. It is not the unit of production. The number that determines whether an AI feature is viable is the fully loaded cost of producing one outcome the business accepts — and that number can move in the opposite direction from the invoice.
This is not an argument against cost optimization. It is an argument for optimizing the right objective. Tokens are a supplier metric: they describe what the provider sold you. A successful task is a product metric: it describes what your customer received. The two are related, but the relationship passes through failure rates, retry behavior, and human recovery, and those are exactly the parts that rarely appear in a vendor dashboard.
The success contract comes before the cost model
You cannot compute cost per successful task until you can say, precisely and testably, what "successful" means. Most teams discover this during their first argument about the metric, usually because two departments are counting different things.
A success contract has five parts:
| Part | Question it answers | Example |
|---|---|---|
| Deliverable | What artifact or state must exist? | Invoice fields persisted to the ledger |
| Acceptance rule | What makes it valid? | All required fields present, totals reconcile, no policy violation |
| Judge | Who or what decides? | Deterministic validator plus spot audit |
| Window | By when must it be accepted? | Within 24 hours of upload |
| Downstream confirmation | What proves it worked? | No correction event within 30 days |
Without the window, a task can be "successful" forever because nobody has complained yet. Without downstream confirmation, you count outputs that pass validation but fail in practice. Without a named judge, success becomes whatever the team reporting the number wants it to be.
Different task types need different contracts, and the differences matter more than the similarity:
- Classification: success is a label that matches an adjudicated gold label, or that survives audit without reversal.
- Extraction: success is a record that passes schema validation, business rules, and a sample audit.
- Generation: success is content accepted with at most minor edits, where "minor" is defined by an edit-distance or field-level threshold, not by vibes.
- Agent action: success is a completed multi-step outcome with no compensating action required and no duplicate side effects.
- Retrieval answer: success is an answer supported by cited sources that a reviewer agrees are sufficient.
Write the contract down before launch, version it, and store the version with each task. When the definition changes, you can still compare periods by recomputing both under the same rule.
A useful test for a contract is whether two people can disagree about a specific case. Take a document where the total is correct but the tax code is missing. Does that pass? If the answer depends on who you ask, the contract is not finished. Add the rule, and add the case to the evaluation set so the next prompt change is measured against it.
The task ledger: one row per task, not per API call
The accounting unit should be the task. A task may issue four model calls, two retrieval queries, one tool call, and one human review. Keying cost by request guarantees you will undercount everything that happens after the first call.
A workable ledger:
create table ai_tasks (
task_id uuid primary key,
task_type text not null,
tenant_id text not null,
success_contract text not null,
created_at timestamptz not null default now(),
completed_at timestamptz,
outcome text not null default 'pending'
check (outcome in ('pending','success','failed','abandoned')),
failure_reason text,
attempt_count integer not null default 0,
model_cost_usd numeric(12,6) not null default 0,
tool_cost_usd numeric(12,6) not null default 0,
infra_cost_usd numeric(12,6) not null default 0,
human_cost_usd numeric(12,6) not null default 0,
escalation_cost_usd numeric(12,6) not null default 0
);
create table ai_task_attempts (
attempt_id bigserial primary key,
task_id uuid not null references ai_tasks(task_id),
model text not null,
prompt_version text not null,
input_tokens integer not null,
cached_tokens integer not null default 0,
output_tokens integer not null,
cost_usd numeric(12,6) not null,
started_at timestamptz not null,
latency_ms integer,
error_code text
);
The ai_tasks row is the denominator. ai_task_attempts explains the numerator. Keeping them separate is what lets you answer "what did this task cost in total?" and "which attempt was the expensive one?" without joining across five systems.
For shared costs — observability, evaluation, platform engineering, vendor minimums — allocate rather than ignore. A simple, defensible rule is cost per task type based on either task volume or a driver such as tokens processed. Pick one, document it, and apply it consistently. Teams that skip allocation end up with a cost per task that excludes the very infrastructure that makes the task possible, then wonder why the margin disappears at scale.
Failures are not free, and some failures are expensive
The most common accounting error is treating a failed task as zero cost. A failed task consumed tokens, possibly tools, possibly a human review, and then consumed the cost of recovery. In a well-run ledger, failure is the most expensive outcome per unit of value because it produces negative value: you paid twice and still need the outcome.
A useful taxonomy separates failures by how they surface:
| Failure type | How it appears | Typical cost profile |
|---|---|---|
| Hard failure | Timeout, 5xx, schema rejection | Low: retry may succeed, little human time |
| Loud wrong output | Validator catches bad values | Medium: rework, sometimes a second model call |
| Silent wrong output | Passes validation, wrong in reality | High: discovered later, often by a human or customer |
| Partial completion | Some fields or steps done, others missing | Medium to high: manual reconciliation |
| Late completion | Correct but after the window | Medium: SLA cost, duplicate work |
| Abandoned | User gives up | High: all cost, no value, plus trust damage |
Silent failures dominate the economics of most production AI features. They are also the ones that a token-based dashboard cannot see. Track them through downstream correction events, customer complaints, and audit reversals — not through error rates, which only capture the loud ones.
The break-even between two models is a quality question
Comparing two models by price per million tokens is straightforward and mostly useless. The useful comparison is cost per success, including rework and escalation.
Let:
c= fully loaded cost per attempt, including tools and allocated infrastructures= probability that the attempt produces an accepted task without extra human workr= expected recovery cost per failed attempt, including human correction and any escalation
Then the expected cost per successful task is:
cost_per_success = (c + (1 - s) × r) / s
For two models A and B, A is cheaper per success when:
(c_A + (1 - s_A) × r_A) / s_A < (c_B + (1 - s_B) × r_B) / s_B
If recovery costs are similar (r_A ≈ r_B = r), the inequality rearranges to a simple floor on the cheaper model's success rate:
s_A > s_B × (c_A + r) / (c_B + r)
The exact algebra is less important than the shape: a cheaper attempt buys you less than the price ratio suggests, because every failure carries a recovery cost that the expensive model avoided. When r is large relative to c, quality dominates price almost completely.
A concrete comparison using current list prices
Take an extraction task with a 3,000-token prompt, 400 output tokens, and no caching in the base case. Prices below are standard list rates as of late 2026; caching and batch discounts would change the absolute numbers but not the ranking logic.
| Model | Input $/M | Output $/M | Cost per attempt |
|---|---|---|---|
| GPT-6 Luna | 0.10 | 0.50 | $0.0005 |
| GPT-6.1 Sol | 2.00 | 10.00 | $0.0100 |
| GPT-6 Astra | 10.00 | 50.00 | $0.0500 |
| Claude Haiku 5.5 | 0.10 | 0.50 | $0.0005 |
| Claude Sonnet 5.5 | 2.00 | 10.00 | $0.0100 |
Now add an operations cost of $9.00 per corrected document — roughly ten minutes at a fully loaded $54 per hour. Assume the mid-tier model succeeds on 96 percent of documents, the cheap model on 84 percent, and the frontier model on 98 percent.
Luna: (0.0005 + 0.16 × 9.00) / 0.84 = $1.7149 per success
Sol: (0.0100 + 0.04 × 9.00) / 0.96 = $0.3854 per success
Astra: (0.0500 + 0.02 × 9.00) / 0.98 = $0.2347 per success
The cheapest model is more than seven times more expensive per accepted document than the most expensive model in this scenario. That is the entire argument for this metric in one table. It also shows why the frontier model can be the rational choice for a task where errors are costly, even though its token price is 100 times higher.
Change the recovery cost to $0.50 — a task where a failure is caught by a validator and retried automatically — and the ranking flips:
Luna: (0.0005 + 0.16 × 0.50) / 0.84 = $0.0958
Sol: (0.0100 + 0.04 × 0.50) / 0.96 = $0.0313
Astra: (0.0500 + 0.02 × 0.50) / 0.98 = $0.0612
Sol still wins, but the gap narrows and Luna becomes defensible if volume is high and the failure is genuinely cheap. The lesson is not "always use the big model". It is that the answer depends on recovery cost, and recovery cost is a property of your workflow, not of the model.
One more subtlety: failures are rarely independent. A prompt that misreads one vendor's invoice format tends to misread every invoice from that vendor, and a retrieval bug tends to affect every task that hits the same document. Averaging a 5 percent failure rate across 100,000 tasks assumes the failures are scattered. If they cluster, the cost arrives in bursts that overwhelm a review queue sized for the average. Model the tail, not just the mean: look at the worst day, the worst tenant, and the worst document class. A model with a slightly higher average failure rate but no clusters can be cheaper in practice than one with a better average and occasional bad afternoons.
Where the numerator quietly grows
Six mechanisms inflate cost per success without showing up in a token dashboard.
Retry amplification. A retry after a malformed response pays for the first generation in full. At a 10 percent malformed rate and two retries, expected attempts per task rise by roughly 11 percent before any quality benefit. If the retry uses a bigger model, the effective price per task is a blend, not the list price of either model.
Context re-sending. Agent loops resend the accumulated transcript on every step. A five-step task can cost three to five times a single-shot call with the same final output. The per-token price did not change; the number of tokens per task did.
Tool fees. Web search at $10 per 1,000 calls, file search at $2.50 per 1,000 calls plus storage, hosted containers billed by size and session. At low volume these are noise. At 200,000 tasks a month they are a line item that must be attributed to the task that triggered it.
Review time. If every task gets a 90-second review, that is 1.5 minutes of labor per task. At a $60 hourly rate, $1.50 per task — often more than the model cost by an order of magnitude.
Escalation. A task escalated to a specialist may consume 20 minutes of expensive time plus the original AI cost. Even a 2 percent escalation rate can dominate the unit economics.
Maintenance. Prompt iteration, evaluation runs, incident response, and vendor migration. Amortize these per task per month. A feature with $400 a month in maintenance and 2,000 tasks a month carries $0.20 per task before a single token is generated.
Discounts that raise cost per success
Caching and batch processing both lower the price of a token. Neither guarantees a lower cost per successful task, and in a few cases they make it worse.
Prompt caching works on an exact prefix match. If the cached prefix contains context that changes slowly — a policy document, a product catalog, a tenant's configuration — you get a large discount and identical behavior. If the cached prefix goes stale, you get the same discount and worse answers. A support assistant that caches yesterday's refund policy will confidently quote yesterday's rules at a fraction of the input price. The invoice improves, the failure rate climbs, and the cost per resolved case can go up even while the per-request cost goes down.
The same logic applies to batch APIs. A 50 percent discount is real, but batch results arrive later. If the success contract has a 24-hour window and the batch completes in 26 hours, you have bought a discount and paid for it with late completions. For back-office work with a weekly SLA, batch is close to free money. For anything a user is waiting on, it usually is not.
The test is the same in both cases: measure the discount and the success rate together, on the same task population, before and after. A 40 percent token reduction with a 5 percentage point increase in silent errors is often a net loss at realistic recovery costs.
Allocating platform cost without a three-week argument
Every AI feature carries shared costs that no single task triggers: observability, evaluation infrastructure, the vector store, the orchestration service, on-call, and the engineer who maintains the prompt registry. Ignoring them produces a flattering number that fails at the first budget review.
Three allocation methods cover most situations.
Per-task allocation. Divide the monthly shared cost by the number of tasks. Simple, stable, and fair when tasks are similar. It breaks down when one task type consumes most of the evaluation effort or storage.
Driver-based allocation. Allocate by a measurable driver: tokens processed, documents stored, evaluation runs, or compute seconds. More accurate, more work to maintain, and prone to disputes about which driver is "right".
Tiered allocation. Assign shared cost to task classes in fixed proportions, reviewed quarterly. Less precise, but stable enough to plan against and cheap to administer.
Whichever you choose, keep two views: contribution margin per task (direct costs only) and fully loaded margin per task (direct plus allocated). The first tells you whether the task is worth doing at all. The second tells you whether the product is worth running. Decisions about pricing need the second; decisions about prompt optimization need the first.
A four-week plan to replace the token dashboard
If you already have a feature in production, you can build this without a migration.
Week 1: define and backfill. Write the success contract for the top three task types. Add a success_contract version column. Backfill outcomes for the last 60 days where possible, using downstream records rather than memory. Expect the backfill to be incomplete; mark it as such and don't mix it with prospectively collected data.
Week 2: capture direct cost. Attach model, prompt version, token usage, tool calls, and latency to each attempt as it happens. Reconcile a single day against the provider invoice by model. If the reconciliation is off by more than 5 percent, find the gap before trusting the rest.
Week 3: sample failures. Pull 30 failures per task type, stratified by failure type. For each, record the human minutes spent, whether the customer noticed, and what the final resolution was. This is the step teams skip, and it is the step that determines whether the metric has any predictive value.
Week 4: compute and stress-test. Produce cost per success for each task type and model variant. Then run the sensitivity analysis: double recovery cost, halve the success rate, and add 20 percent to the attempt cost. If the preferred option changes under any of these, the decision needs more data, not more confidence.
After week four, put the number on a monthly cadence with a fixed definition. Review it alongside the quality metrics, not in a separate meeting. The point is to make cost and quality one conversation, because they are one system.
The denominator is where teams cheat, often unintentionally
Every organization that reports cost per successful task eventually faces pressure to improve the number. There are four common ways the denominator gets softened. All of them are easy to do without noticing.
Counting attempts instead of tasks. A task that took three attempts counts as one success and three entries in the cost numerator. If the dashboard divides by attempts, the cost per "task" falls as retries increase.
Redefining success downward. "Accepted with edits" becomes success. Then "accepted after a second model pass" becomes success. Each redefinition is defensible in isolation and collectively they make the metric incomparable over time.
Excluding escalations. Escalated tasks are removed from both numerator and denominator because they are "handled by a different team". The cost does not disappear; it moves to a cost center that is not in the report.
Truncating the attribution window. If success is measured at 24 hours, late failures never register. A 30-day downstream correction rate is often two to four times the 24-hour rate for document workflows.
Guard against all four with a written definition, a versioned contract, and a monthly reconciliation between the ledger and the finance system. If the two disagree by more than a few percent, find out why before presenting either number.
Instrumentation that produces a trustworthy number
The minimum viable measurement is an event per task transition, with a stable task ID and the cost components attached.
from dataclasses import dataclass, field
from decimal import Decimal
@dataclass
class TaskCost:
task_id: str
attempts: int = 0
model_cost: Decimal = Decimal("0")
tool_cost: Decimal = Decimal("0")
human_minutes: Decimal = Decimal("0")
escalation_cost: Decimal = Decimal("0")
accepted: bool = False
def total_cost(self, loaded_rate_per_minute: Decimal) -> Decimal:
labor = self.human_minutes * loaded_rate_per_minute
return self.model_cost + self.tool_cost + labor + self.escalation_cost
def cost_per_success(tasks, loaded_rate_per_minute):
successes = [t for t in tasks if t.accepted]
if not successes:
return None
spend = sum((t.total_cost(loaded_rate_per_minute) for t in tasks), Decimal("0"))
return spend / len(successes)
Note the denominator: accepted tasks. Note the numerator: spend on all tasks, including failures. That asymmetry is the whole point. If you only sum costs for successful tasks, failures become free and the metric becomes a measure of how much you spent on things that worked, which is not a business question.
Emit success from the system that owns acceptance, not from the AI service. The AI service knows it produced output; only the downstream system knows the output was used. For agent workflows, add a reconciliation job that looks for compensating actions, duplicate records, and manual corrections within the window.
Cost per success is also a pricing input
Once you have the number, it becomes useful outside engineering. A per-task cost of $0.39 against a plan that includes 500 tasks a month at $49 contributes $195 in cost against $49 in revenue — a loss that no amount of token optimization fixes. Conversely, a $0.04 cost per task supports a generous free tier that would look reckless under token-based accounting.
Two derived numbers matter for product decisions:
gross margin per task = price per task - cost per successful task
break-even success rate = (c + r) / (price + r)
The second formula is a sanity check on quality investment. If the price per task is $0.50, the attempt cost is $0.02, and the recovery cost is $9, you need a success rate above roughly 95 percent for the task to be margin-positive. That is a much more useful target for an evaluation program than "improve accuracy".
A decision framework you can apply this week
Use this sequence when comparing models, vendors, or architectures.
- Write the success contract. Deliverable, acceptance rule, judge, window, downstream confirmation. Version it.
- Build the task ledger. One row per task, attempts in a child table, costs attributed at write time.
- Measure recovery cost per failure type. Sample at least 30 real failures. Include human minutes, escalation, and customer-visible impact.
- Compute cost per success, not per attempt. Include failures in the numerator, exclude them from the denominator.
- Run the comparison at your actual volume. Tool fees, cache hit rates, and batch discounts change with scale.
- Stress the assumptions. Double the recovery cost and halve the success rate. If the ranking flips, you don't have a decision yet.
- Recheck monthly. Prompts, models, and traffic mix drift. A cost-per-success number older than a quarter is a historical artifact.
- Publish the definition with the number. A metric without its contract is an invitation to argue about definitions instead of improving the system.
The cheapest token is not the cheapest task. The cheapest task is the one that gets accepted once, by the system that needed it, without a human repairing it afterward. Price the whole path, and the model choice usually makes itself.