A support ticket classifier goes live with 91% agreement against a hand-labeled sample. For five months it looks fine: no errors, no alerts, stable latency. Then someone notices that the "billing" queue has grown 30% while "account access" has shrunk, and that a new plan tier launched in March is generating tickets the model files as "other." Nothing in the pipeline failed. The model is doing exactly what it was asked, on inputs that no longer resemble the ones it was evaluated on.
That's drift, and for AI-generated classifications it's the default state rather than an exception. Products change, customers change, the language people use changes, and the model behind your API changes whether or not you do. The classifier that was right in January can be quietly wrong in July while every operational metric stays green.
This article is about detecting that before a human notices. It covers the four kinds of drift that matter for LLM classifiers, the signals you can monitor without labeled data, the statistics worth using (and their limits), and a practical monitoring loop that a small team can actually run.
Four kinds of drift, only one of which is usually monitored
"Drift" gets used loosely. For a classification feature, it's worth separating four phenomena, because they require different responses.
1. Input drift
The distribution of what arrives changes. New product names, new customer segments, a new language, longer messages, a new attachment type. The model's decision boundary hasn't moved, but it's now being asked questions from a region of input space it saw rarely during evaluation.
Input drift is the easiest to detect because you can measure it without labels. It's also the most common cause of downstream label shifts.
2. Output or label drift
The distribution of predicted labels changes. "Billing" goes from 22% to 34% of volume. This is what the opening scenario noticed. It can be caused by input drift, by a model change, or by a threshold change, so it's a symptom rather than a diagnosis.
3. Concept drift
The correct answer for the same input changes. A policy update means that a ticket about a refund window is now "billing" rather than "policy." The input looks identical; the label should differ. This is the hardest to detect because no amount of input monitoring reveals it, and output distribution may not move much if the change affects a small slice.
4. System drift
Your prompt changed, the provider updated the model behind an alias, the retrieval index was rebuilt, a preprocessing step started stripping signatures, or the label taxonomy was edited. The model's behavior changed for reasons that have nothing to do with the world.
In LLM systems, system drift is often the dominant cause of sudden regressions, and it's the one teams are least prepared for because it originates inside their own stack or their vendor's.
⚠️ Gotcha: If you call a moving model alias rather than a dated snapshot, the provider can change your classifier's behavior without any deploy on your side. That is system drift, and your change log will be empty.
Why accuracy monitoring doesn't save you
The textbook answer is to monitor accuracy against ground truth. For most production classifiers, ground truth doesn't arrive in time to be useful, or doesn't arrive at all.
Three problems recur:
Labels are delayed. A ticket's correct category may only be known after it's resolved, which can be days. A refund classification might be confirmable only when finance processes it. By the time you have labels, the drift has been live for a week.
Labels are expensive. Human review of a random sample is the gold standard and the first thing cut when the team is busy. A 200-item weekly audit is 200 judgments, plus adjudication, plus the tooling to route them.
Review is biased. Most teams don't sample randomly. They review low-confidence outputs, customer complaints, and escalations. That sample tells you about the cases that were already visible. It systematically misses the confident wrong answer, which is the one that does damage.
So label-based accuracy is necessary but insufficient. You need leading indicators that don't require labels, plus a small, deliberately stratified audit to anchor them.
Signals you can compute without labels
These are cheap, available on every request, and collectively good at catching change.
Predicted label distribution
Track the proportion of each class over time, with confidence intervals. A shift beyond normal variation is a signal. For a classifier with ten classes and 5,000 requests a day, even a few percentage points of movement is statistically detectable, which is why you need a practical threshold as well as a statistical one.
Prediction entropy and top-two margin
If the model returns probabilities or logprobs, track the entropy of the distribution per request and the margin between the top two classes. Rising entropy or shrinking margins means the model is less sure, which often precedes a label shift and correlates with errors. If you only get a label, you can approximate this with repeated sampling at non-zero temperature, but that costs money; use it on a sample.
Parse and schema failure rate
Every increase in malformed JSON, missing fields, or out-of-vocabulary labels is a system-drift signal. A provider change or prompt edit often shows up here first.
Refusal and fallback rate
LLM classifiers sometimes refuse, return an empty string, or emit "I can't determine this." Track that rate separately from the "other" class. A refusal spike is a strong indicator that inputs have moved outside the model's comfort zone, or that a safety layer is now triggering on benign content.
Evidence and citation rate
If the classifier is supposed to cite the fields or passages it used, track how often it does. A drop means it's guessing.
Output length and structure
For extraction-style classifiers, changes in the number of extracted entities or the length of generated rationale are useful proxies. They're noisy, but a step change is informative.
Disagreement with a frozen reference
Keep a small, frozen model or a rules baseline and measure agreement on a sample of live traffic. You don't need the reference to be better; you need it to be stable. A sudden drop in agreement means one of the two changed, and the frozen one didn't.
Embedding-space occupancy
If you embed inputs for retrieval or clustering, track the share of requests that fall far from the reference centroid, or the emergence of new clusters. This catches new intents before they have a name.
Input-side features
Length, language, source channel, attachment presence, customer segment, and the rate of unseen tokens or product names. These are often the earliest signal, and they're free.
The statistics, and where they mislead
For distribution comparison, a handful of methods cover most needs.
| Method | Works on | Output | Practical note |
|---|---|---|---|
| Population Stability Index (PSI) | Numeric or categorical | Non-negative score | Common thresholds: <0.1 stable, 0.1–0.2 moderate, ≥0.2 significant. Insensitive to small shifts by design. |
| Kullback–Leibler divergence | Distributions | Non-negative score | Asymmetric; undefined when a bin has zero mass in the reference. Smooth first. |
| Jensen–Shannon distance | Distributions | 0–1 (for log base 2) | Symmetric and bounded, easier to threshold across classes. |
| Chi-square test | Categorical counts | p-value | Sensitive to large samples; a trivial difference becomes "significant." |
| Kolmogorov–Smirnov | Continuous | p-value / statistic | Sensitive to sample size and to the middle of the distribution. |
Evidently's documentation is a useful reference for defaults and thresholds; it uses PSI and Jensen–Shannon as defaults for tabular columns and applies Wasserstein distance to numeric columns above roughly a thousand observations (Evidently docs). The library is not the point; the conventions are.
Two warnings matter more than the choice of test.
Significance is not importance. With 50,000 requests a day, almost any change will produce a p-value below 0.05. A classifier can be "significantly different" while being operationally identical. Set thresholds in terms of impact: how many items change class, how much review capacity the change consumes, how much money is misrouted.
Small segments hide. Aggregate drift can be flat while one customer segment, language, or document type has moved drastically. Monitor the segments that matter commercially, and accept that small segments need longer windows before a signal is trustworthy.
A simple PSI implementation with reference-derived bins is enough for a daily job:
import numpy as np
import pandas as pd
def psi(reference: pd.Series, current: pd.Series, bins: int = 10) -> float:
ref = reference.dropna()
cur = current.dropna()
if ref.empty or cur.empty:
return float("nan")
edges = np.quantile(ref, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
edges = np.unique(edges)
if len(edges) < 3:
return 0.0
ref_pct = np.histogram(ref, bins=edges)[0] / len(ref)
cur_pct = np.histogram(cur, bins=edges)[0] / len(cur)
eps = 1e-6
ref_pct = np.clip(ref_pct, eps, None)
cur_pct = np.clip(cur_pct, eps, None)
return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))
For label proportions, a chi-square on counts is fine, but I'd pair it with a simple effect size: the maximum absolute change in class share. That number is what a support lead can act on.
Detecting concept drift when labels are scarce
Concept drift is the case where the input is unchanged but the correct label isn't. You can't see it in input distributions, and you often can't see it in output distributions either.
Three approaches work in practice.
Targeted audits around known change events. When a policy changes, a plan is renamed, or a product is deprecated, run a focused audit on the affected cases rather than waiting for the monthly sample. The event tells you where to look. This is the highest-yield form of human review.
Stratified sampling by predicted class and uncertainty. Draw a fixed number from each class, plus a sample from the high-uncertainty tail. This catches class-specific concept drift that random sampling would underweight, and it gives you per-class accuracy estimates with reasonable variance.
Delayed-label joins. If ground truth arrives later through a downstream system — a refund processed, a ticket reopened, an invoice approved — join it back to the original prediction. This is the most reliable signal, and building the join is usually a day of work that pays for itself.
When you do audit, measure agreement between annotators on a subset. If two humans disagree on 15% of "billing vs account" cases, the classifier's 91% agreement is partly measuring label noise, and a drift alert on that boundary is not actionable.
A monitoring loop that fits in a small team
The failure mode of drift monitoring is building an elaborate dashboard nobody reads. A minimal loop with clear cadence is better.
Daily: canary run. Execute the frozen golden set (200–500 items) against the current production prompt and model at temperature 0. Record exact-match or macro-F1. Alert on a drop beyond a threshold you've chosen from observed variance. This is essentially a regression test for the model, and it catches system drift within a day.
Daily: distribution snapshot. Store per-class counts, refusal counts, parse failures, and a few input features. Compare the last seven days to a reference window with PSI or a share-delta. Alert only when the effect exceeds a practical threshold.
Weekly: stratified audit. Sample 50–100 items across classes and uncertainty bands, label them, and compare. Track per-class precision and recall. This is where human judgment actually enters, and it's small enough to sustain.
Monthly: segment review. Break the metrics down by tenant tier, language, channel, and document type. Look for segments that are diverging even when the aggregate is stable.
Quarterly: taxonomy and reference review. Re-read the label definitions against how the business now talks. Decide whether to re-baseline. A taxonomy change is a concept change for the model, and it should be treated as a release, not an edit.
The canary run deserves emphasis. It's the cheapest defense against system drift: fixed inputs, fixed expected outputs, run on a schedule, alert on change. It won't catch input drift, because the inputs are fixed. It will catch the model or prompt changing underneath you, which is the failure that otherwise has no symptom until users complain.
What to do when an alert fires
An alert that says "drift detected" is not actionable. The response should follow a short decision path.
- Confirm the signal. Is it a real distribution change, or a logging gap, a timezone boundary, a batch import, or a holiday? Check volume and missingness first.
- Localize it. Which classes, segments, and input features moved? A shift confined to one tenant's document format is a different problem from a global shift.
- Attribute it. Compare against the change log: prompt deploys, model version, retrieval index rebuilds, upstream schema changes, product launches, marketing campaigns.
- Sample and read. Pull twenty items from the shifted region. Reading real inputs is faster than any dashboard for forming a hypothesis.
- Decide. Three options: fix the cause, re-baseline the monitor, or accept the new distribution and adjust capacity. Write down which one and why.
- Verify. After the fix, confirm the metric returns to range. If it doesn't, the cause was misidentified.
The most common mistake in step five is re-baselining to silence an alert. Re-baselining is legitimate when the world genuinely changed and the new distribution is correct — a new product line, a new customer segment. It is not legitimate when the model got worse. Distinguish them with the audit, not with the dashboard.
A note on multi-label and extraction tasks
Single-label classification is the easy case. Multi-label and extraction need slightly different signals.
For multi-label, track the distribution of labels per item (how many labels, which co-occur), not just per-label counts. A model that starts applying one label to everything will show a sharp drop in label-count diversity.
For extraction, monitor field-level fill rates, value distributions, and validation failure rates. "Total amount extracted" should have a distribution; if it suddenly clusters at zero, you have a parsing problem. Also monitor the relationship between extracted fields — if line items no longer sum to the total, either the extraction or the source changed.
In both cases, the frozen canary still applies, and the stratified audit still applies. The metrics change; the loop doesn't.
A worked example: the plan-tier drift
Suppose a B2B support classifier with six labels: billing, account access, integrations, data export, bug report, and other. Baseline volume is 6,000 tickets a day, and the launch evaluation showed 91% agreement on a 400-item sample.
In March, the company launches a usage-based plan. By May, the following is true but invisible:
- The share of tickets mentioning "credits" rises from 1.2% to 9%.
- The "billing" share rises from 22% to 29%.
- The "other" share rises from 6% to 8%.
- Parse failures and refusals are flat.
- Canary scores are flat, because the golden set contains no usage-based-plan tickets.
Every operational dashboard is green. The canary is doing its job — the model and prompt haven't changed — but the canary can't see the world change. The label distribution is the only signal, and it's sitting in a table nobody graphs.
When you break the shift down by segment, the pattern is sharper. Tickets from accounts on the new plan account for almost all of the "billing" increase, and within that segment, 40% of tickets are labeled "billing" with an extracted amount of zero. The model has learned that plan language is billing language, and it's applying that association to questions about credits that are actually account-access issues.
The response has three parts. First, a taxonomy decision: is "credits" its own class, or a sub-label of billing? That's a product question, not a model question. Second, a labeled audit of 100 new-plan tickets to measure actual accuracy in the shifted segment, which turns out to be 71% — well below the aggregate. Third, a golden-set expansion with 60 representative new-plan cases, so the next change is evaluated against reality.
Note what did the work: a distribution chart, a segment breakdown, and a human reading tickets. Not a sophisticated drift detector. The statistics pointed at the region; the audit explained it.
How many items do you need to sample?
Sample size determines what you can detect, and teams routinely sample too little to see anything but a catastrophe.
For a proportion, the standard error is roughly sqrt(p(1-p)/n). With 100 samples and a 20% error rate, the 95% interval is about ±8 points. That's wide enough to hide a meaningful regression. With 400 samples, it's about ±4 points. With 1,000, about ±2.5.
This has two consequences. First, a weekly audit of 50 items can only detect large changes; don't over-interpret small movements in it. Second, if you need to detect a 3-point change in a specific class, random sampling from the whole stream is inefficient. Stratify: take a fixed number per class, then weight back to the true distribution when computing aggregate accuracy.
Stratified sampling also solves the rare-class problem. If "data export" is 3% of volume, a 100-item random sample contains three examples on average. You can't estimate its accuracy from that, and you certainly can't detect drift in it. Drawing 25 per class gives you a usable estimate for every class at the cost of 150 labels.
The same logic applies to uncertainty bands. If the model exposes a confidence signal, sample the bottom decile deliberately. Errors concentrate there, so a small sample from that band finds problems faster than a large random one.
The reference window problem
Every drift metric compares current data to a reference. Choosing the reference is a modeling decision with consequences.
Fixed reference from launch. Stable and easy to explain. It will eventually flag legitimate change as drift, because the world moves. Useful for detecting the cumulative distance from the system you validated.
Rolling reference (e.g., previous 28 days). Adapts to gradual change and flags sudden movement. It can also "learn" a regression: if quality degrades slowly over two months, the rolling reference follows it down and stops alerting.
Seasonal reference (same period last year or last quarter). Handles weekly and annual cycles. Needs at least a year of history and enough volume per period.
The practical answer is usually two references: a fixed launch baseline for the canary and long-horizon reporting, and a rolling window for operational alerts. When the rolling monitor fires, compare against the fixed baseline to see whether this is a new excursion or the cumulative result of a long slide.
Re-baselining should be an explicit, recorded event with a reason: "new plan launched, distribution change expected, baseline updated 2026-05-01." Without that record, the next person can't tell whether a flat chart means stability or a series of quiet resets.
The uncomfortable truth about thresholds
Every threshold is a trade-off between alert sensitivity and alert fatigue. A threshold tight enough to catch a 3% shift will fire on normal weekly variation. A threshold loose enough to be quiet will miss the slow drift that matters most.
The way out is to monitor two things at different sensitivities: a fast, sensitive signal for step changes (canary failure, parse errors, refusal spike) and a slower, high-effect signal for gradual drift (weekly PSI with a practical threshold). Step changes should page someone. Gradual drift should create a ticket.
That division matches how the failures actually present. A provider change is a step. A new customer segment is a ramp. Treating them with the same alert policy guarantees that one is too noisy and the other is too slow.
Drift detection isn't a model you install once. It's a small set of stable measurements, a cadence, and a habit of reading real inputs when something moves. The classifier doesn't need to be perfect; the monitoring does need to tell you when it stopped being right.