The triage assistant shipped in March. Median handle time on support tickets fell from 11 minutes to 7. The dashboard celebrated a 36 percent improvement. Six weeks later the backlog was larger, the team had hired a second reviewer, and customers were complaining that they had to explain their problem twice.
Nothing in the dashboard was wrong. Handle time really did fall. What the metric missed was that four minutes had not disappeared — they had been relocated. An agent now spent roughly 90 seconds reading and correcting the AI's draft, another 40 seconds chasing a missing order ID the model had invented, and the customer spent two minutes re-stating context that the assistant had dropped between turns. The work moved from the person doing the task to three other places, none of which appeared in the original measurement.
This is the default failure mode of AI time savings. Time is easy to measure at the point where you removed it and easy to ignore everywhere it reappears. If you want to know whether an AI feature is a net gain, you have to follow the minutes.
The metric that flatters everyone: "time saved per task"
Most teams start with a simple subtraction. They time the old workflow, time the new one, and multiply the difference by volume. It produces a number that executives like and that almost never survives contact with a P&L.
The arithmetic has four problems.
First, the old workflow's duration is usually recalled, not measured. People remember the interesting cases and forget the waiting. Second, the new workflow's duration is measured at the wrong boundary — often from "user clicks generate" to "response appears", which excludes the prompt cleanup before and the verification after. Third, it assumes the saved time converts into output. In practice it often converts into idle time, extra polishing, or a larger queue elsewhere. Fourth, it ignores rework. An AI draft that gets rewritten from scratch did not save the writing time; it added a reading time.
A defensible measurement starts with a different question: where did the minutes go, and who is holding them now?
Hands-on time, elapsed time, and calendar time are three different numbers
Before measuring anything, fix the vocabulary. Teams routinely compare one kind of time against another and conclude something false.
| Measure | Definition | What it answers |
|---|---|---|
| Touch time | Minutes a human is actively engaged: reading, typing, deciding, correcting | Did the workload shrink? |
| Elapsed time | Wall-clock duration from request start to completion | Did the customer wait less? |
| Calendar time | Duration including waits for batches, approvals, or business hours | Did the process get faster end to end? |
| Queue time | Time a task spends waiting for a human or a system | Did we just add a waiting room? |
| Rework time | Minutes spent redoing work that failed a check or a review | Is the output actually usable? |
An AI feature can improve one and damage another. A drafting tool might cut touch time by 40 percent while adding a 20-minute review queue, leaving elapsed time unchanged. A classifier might reduce elapsed time sharply while increasing rework time, because the 8 percent of cases it gets wrong now take three times as long to unwind.
The number your finance team cares about is touch time plus rework time, divided by accepted output. The number your customers care about is calendar time. The number your operations lead cares about is queue time and work in progress. You need all three, and you need them at the same task boundary.
Where the work actually goes
In practice, relocated work lands in five places. Naming them makes them measurable.
Review and verification. Someone reads the output to decide whether it is safe. This is the most visible destination and the one teams usually budget for. It is also the one most often underestimated, because reviewers are often senior people whose minutes cost more than the minutes that were "saved".
Correction and rework. The output is wrong in a way that requires a human to fix it, or requires a second AI attempt with a better prompt. Retries are cheap in tokens and expensive in attention.
Context gathering. The model lacks information, so a human goes to find it: the customer's plan, the prior ticket, the current price, the exception in the policy. If the AI feature does not fetch that context automatically, the human becomes the retrieval layer.
Customer or colleague effort. The user re-enters data, repeats themselves, or contacts support because the automated path failed. This is real labor even when it is unpaid.
Maintenance. Prompt tuning, evaluation sets, incident review, and the slow accumulation of special cases. This work is invisible at the task level and large at the quarter level.
🔍 A useful test: for each bucket, ask "who is doing this now, and what were they doing before?" If the answer is "the same person, and they were doing the task", you have not saved time — you have changed its shape.
The five work-location buckets, with examples
Consider a feature that summarizes long customer emails for a sales team. The demo shows a 900-word email becoming a four-line summary in three seconds. The time study should look like this:
| Bucket | Before | After |
|---|---|---|
| Reading | 4.5 min per email | 1.5 min reading summary + 0.5 min spot-checking the original |
| Writing notes | 2 min | 0.5 min editing the summary into CRM fields |
| Correction | occasional | 12% of summaries omit a commitment or a date, requiring a re-read |
| Context gathering | included in reading | 0.7 min finding the account record the summary did not link |
| Maintenance | none | prompt iteration, evaluation, and a weekly review of flagged summaries |
The gross saving is 4.5 minutes. The net saving is closer to 1.8 minutes once correction and context gathering are counted, and it disappears entirely if the summary has to be verified line by line for a regulated process.
That is still a real gain. The point is not that AI never saves time. The point is that the gross number and the net number differ by a factor that determines whether the feature is worth running.
Self-reported savings are systematically optimistic
Ask people how much time a tool saves and you will get a number shaped by four biases.
Recall bias. The old workflow is remembered as a blur of competent effort, not as a sequence of specific waits.
Selection bias. People remember the impressive saves and forget the ordinary cases where the tool was slower than doing it themselves.
Attribution bias. When the AI helps, the success feels like the user's judgment. When it hinders, the hindrance is attributed to the tool. Both attributions inflate the tool's perceived value in different directions, and they don't cancel.
Double counting. A manager reports that the team saves two hours a day. The team reports that they save an hour. Both are estimating the same hour.
Self-reports are still useful, but as a diagnostic, not a metric. Ask people to describe the last five tasks in sequence rather than to estimate an average. Specific episodes surface the hidden steps; averages hide them.
Measure the workflow, not the model call
The single most important instrumentation decision is to put the task — not the API request — at the center of the data model. One task may involve several model calls, a retrieval step, a human review, and a correction loop. If your logs are keyed by request, you cannot compute the cost or the time of the thing the business cares about.
A minimal event schema:
create table task_events (
event_id bigserial primary key,
task_id uuid not null,
task_type text not null,
actor_type text not null check (actor_type in ('human','ai','system')),
actor_id text,
event_name text not null,
occurred_at timestamptz not null,
duration_ms integer,
metadata jsonb not null default '{}'::jsonb
);
create index on task_events (task_id, occurred_at);
create index on task_events (task_type, occurred_at);
Events worth emitting from day one: task_created, ai_draft_requested, ai_draft_returned, human_review_started, human_review_finished, edit_saved, correction_requested, escalated, task_completed, task_reopened. Each human event should carry the actor, so you can see who absorbed the work.
With that table, touch time is the sum of intervals where a human was active, which is not the same as the span between the first and last human event. If a reviewer opens a task, goes to lunch, and comes back, the span is 70 minutes and the touch time is six. Instrument start and stop explicitly, or infer it with an idle threshold and state the assumption.
from datetime import timedelta
IDLE_CUTOFF = timedelta(minutes=5)
def touch_time(events, idle_cutoff=IDLE_CUTOFF):
"""Sum active human intervals, splitting on gaps longer than idle_cutoff."""
human = [e for e in events if e["actor_type"] == "human"]
human.sort(key=lambda e: e["occurred_at"])
total = timedelta(0)
active_since = None
last_seen = None
for event in human:
ts = event["occurred_at"]
if active_since is None:
active_since = ts
elif last_seen is not None and ts - last_seen > idle_cutoff:
total += last_seen - active_since
active_since = ts
last_seen = ts
if active_since is not None and last_seen is not None:
total += last_seen - active_since
return total
This is deliberately simple. It will overstate touch time for tasks where a human is thinking without producing an event and understate it where events are missing. Both errors are visible if you plot the distribution and compare a sample against screen recordings or time diaries. Do that comparison on fifty tasks before trusting the dashboard.
The scorecard: eight numbers that survive a skeptical review
A time-saving claim is credible when it is supported by a small set of measures taken at the same boundary, before and after, on comparable work.
| Metric | Formula | Why it matters |
|---|---|---|
| Net touch minutes per accepted task | (human active minutes + rework minutes) / accepted tasks | The labor actually consumed |
| First-pass acceptance | tasks accepted without edits / tasks submitted | Whether the AI output is usable |
| Rework rate | tasks requiring a second attempt or manual correction / tasks | Hidden cost of wrong output |
| Review minutes per task | reviewer active minutes / reviewed tasks | The price of oversight |
| Elapsed p50 and p90 | request start to completion | What the customer experiences |
| Queue wait p90 | time waiting for human or system | Whether you built a bottleneck |
| Throughput per FTE | accepted tasks per person per day | Whether saved time became output |
| Escalation rate | tasks handed to a human specialist / tasks | Where complexity lands |
Two derived measures are worth putting on the same page:
net time saved per task = baseline net touch minutes - current net touch minutes
realized capacity = net time saved per task × accepted tasks per day
If realized capacity is positive but throughput is flat, the constraint is somewhere else. That is not a failure of the AI feature; it is a failure to act on the result. A team that saves 90 minutes a day and spends them on a backlog of low-value work has saved time and gained nothing.
A worked example, with the uncomfortable arithmetic
A claims team processes 600 documents a day. The baseline is 8.5 minutes of touch time per document, of which 5 minutes is reading and keying fields, 2 minutes is checking against policy, and 1.5 minutes is fixing exceptions.
The AI extractor reduces reading and keying to 2.5 minutes. It also produces a confidence score. Documents below the threshold go to a reviewer, who spends 3 minutes each. The review rate is 35 percent. Twelve percent of all documents need a correction after review, costing another 4 minutes each.
before: 8.5 min × 600 = 5,100 min/day
after:
auto-accepted (65%): 2.5 min × 390 = 975 min
reviewed (35%): 3.0 min × 210 = 630 min
corrections (12%): 4.0 min × 72 = 288 min
base touch on reviewed docs already counted
--------
1,893 min/day
net saving: 3,207 min/day, or 5.3 min per document
That is a 63 percent reduction, which is genuinely large. Now add the maintenance load: 6 hours a week of prompt and evaluation work, plus 4 hours a week of incident review, is 10 hours, or 120 minutes per working day. Net saving falls to 3,087 minutes — still large, but now the number has a denominator that includes the cost of keeping the feature alive.
The more interesting question is what happens at the margin. If review capacity is 180 documents a day and review demand is 210, the queue grows by 30 documents a day. Within a week the elapsed time for every document is worse than the baseline, even though touch time is better. The team will experience the feature as slower, and they will be right.
The counterfactual problem: what would have happened without the feature?
Every pre/post comparison assumes nothing else changed. In real teams, three things usually did: the work mix shifted, the team changed, and the process changed because the feature made some steps easier.
Three designs are more credible than a naive before/after.
Holdout. Keep 5 to 10 percent of eligible tasks on the old path for a fixed period. Compare net touch time and quality on comparable work. This is the cleanest design and the hardest to sustain politically, because someone has to explain why a customer got the slow path.
Stepped rollout. Roll the feature out team by team or region by region, in a fixed order. Compare each newly treated group against groups not yet treated, controlling for time. This works well when the rollout is already staged for operational reasons.
Matched comparison. When randomization is impossible, match tasks on observable characteristics — type, length, customer tier, prior complexity — and compare within matched pairs. It is weaker than randomization, but far better than comparing March's average to June's average.
Whichever design you use, pre-register the primary metric and the analysis window. The most common analytical failure is not statistical; it is choosing the metric after seeing the data.
Distributions, not averages
Task durations are skewed. A mean of 7 minutes can hide a median of 4 and a p95 of 35. The p95 is where the failures live, and it is where users form their opinion of the feature.
Report the median and the 90th percentile, plus the share of tasks above a service-level threshold. If you must report a mean, report the standard error next to it. For a sample of 200 tasks with a standard deviation of 6 minutes, the 95 percent confidence interval around a mean saving of 1.5 minutes is roughly ±0.8 minutes — wide enough that a "small" win may be indistinguishable from noise.
Use a paired analysis when the same task types appear before and after. If you can't pair, bootstrap the difference in medians rather than assuming normality. The code is short and the result is far more honest than a t-test on a log-normal variable.
Not all saved minutes are worth the same
A minute of a specialist's time and a minute of a junior associate's time both show up as one minute in a time study. They do not show up as one unit of cost, and they definitely do not show up as one unit of scarce capacity.
Weight the minutes before you total them. A simple weighted touch time is:
weighted touch minutes = Σ (minutes by role × role cost rate)
Using fully loaded hourly rates keeps the comparison honest, but the more useful weight for capacity planning is scarcity: how hard is it to add another hour of this person's attention? A reviewer who is also the only person who understands the pricing exceptions is a different constraint from a reviewer who can be trained in a week.
This changes the conclusion in a specific and common way. A feature that saves ten minutes of junior processing time and adds three minutes of senior review time can be a net loss even when the raw minutes fall. The dashboard shows a seven-minute win; the cost line shows a loss; the queue shows a bottleneck. All three are true, and only one of them will be noticed.
There is a second asymmetry: interruption cost. Thirty seconds of review inserted into a specialist's flow is not thirty seconds. It is thirty seconds plus the time to reload the mental state of whatever they were doing. If the review arrives as an interrupt, the effective cost can be several times the measured duration. Batch the reviews, or route them to a role whose job is review, and the same nominal minutes become cheaper.
A time-diary protocol people will actually complete
Instrumentation catches what happens inside your application. It misses work that happens in other tools: the CRM tab, the spreadsheet, the phone call, the Slack thread asking whether the customer is on the new plan. For that, you need a short, low-friction diary.
The protocol that works in practice is not "track your time". It is a five-question sample on a small number of tasks:
- What task were you working on, and what was its ID?
- When did you start actively working on it, and when did you stop?
- How many minutes did you spend outside the tool — searching, asking, checking another system?
- Did you change, redo, or discard anything the AI produced? Roughly how long did that take?
- Was the output usable as-is, usable after edits, or not usable?
Ask for four tasks a day, chosen by a random prompt from the tool, for two weeks. Fifty tasks per role is enough to see the shape of the distribution and to calibrate the automated events. Do not ask for a week at the end of the quarter; the recall will be a fiction.
Two calibration checks make the diary credible. First, compare diary durations with event-derived durations for the same task IDs and report the gap. Second, have a second person independently code a handful of tasks from screen recordings, and compute agreement on where the time went. If agreement is poor, your categories are ambiguous, not your people.
The diary's real value is not precision. It is discovering the buckets you forgot to instrument.
When the saving is real but the organization can't see it
Sometimes the measurement is correct and the feature still looks useless. Four structural reasons explain most of these cases.
The constraint moved. If review is now the bottleneck, saving five minutes of preparation per task only lengthens the queue. Throughput is set by the slowest step.
Batch sizes grew. People wait to accumulate a batch before reviewing, because switching costs make small batches inefficient. Average wait rises even when total work falls.
Saved time is fragmented. Twenty 30-second savings scattered across a day don't become a free hour. They become twenty context switches. Calendar-level gains require blocking work, not just reducing it.
The work was never the bottleneck. If the team was waiting on customers, regulators, or another department, making the internal step faster changes nothing observable. Measure the process, not the step.
The fix is usually operational rather than technical: rebalance review capacity, change batch policy, or redefine who owns the downstream step. An AI feature that saves time inside a badly designed process mostly makes the bad design cheaper to run.
What to instrument before you need the answer
The teams that measure this well instrument first and ask questions later. A practical minimum:
- A stable task ID that survives retries, reviews, and escalations.
- Human start and stop events, not just page views.
- An explicit
correctionevent with a reason code. - A review queue timestamp at enqueue and dequeue.
- The AI's own latency and token usage, attached to the task.
- A flag for whether the task was auto-accepted or human-approved.
- A weekly sample of tasks reviewed against screen recordings or diaries.
- A baseline captured before launch, on the same task definitions.
The last one is the one teams skip. Without a pre-launch baseline, every later number is a comparison against a memory. If the feature is already live, capture a baseline from the holdout group or from the least-treated segment, and label it as reconstructed.
A decision framework for the next time study
Use this when someone asks whether the assistant is worth keeping.
Step 1: Define the unit of work in business terms. "A claim document with all fields extracted and accepted" is a unit. "An API request" is not.
Step 2: Draw the task boundary. Start when the work enters the team's ownership. End when it is accepted downstream. Include every human touch inside that boundary.
Step 3: Count minutes by actor. Human, AI, and system time separately. AI time matters for latency, human time for cost.
Step 4: Add the rework loop. Track tasks that come back, and count the second pass in full.
Step 5: Compare net touch time per accepted task, not per attempt. Failures should raise the cost of success, not disappear from the denominator.
Step 6: Check throughput and queue time. If touch time fell and nothing else moved, find the constraint before claiming a win.
Step 7: Subtract maintenance. Prompt work, evaluation, and incident response are part of the feature's cost. Amortize them per task.
Step 8: State the counterfactual. Holdout, stepped rollout, or matched comparison. Say which, and say what it cannot rule out.
If the net number is positive, the feature is doing what it claims. If it is positive but small, the honest conclusion is that the feature is a convenience, not a capacity gain — and that is a perfectly good reason to keep it, as long as nobody is planning headcount around it.
The trap is not that AI moves work around. All tooling moves work around; that is what a tool is. The trap is measuring only the part that moved off the old path and calling the difference savings. Follow the minutes to their new address, and the decision becomes obvious.