The first version of a custom AI pipeline is usually built in a week. A queue, a worker, a prompt file, a call to a provider, a JSON parser, and a table to store results. It works. The demo lands. Everyone moves on to the next feature.
Eighteen months later, the same pipeline has four prompt variants, two model providers, a retry policy nobody remembers choosing, an evaluation script that only runs on one engineer's laptop, and a nightly job that fails silently when a vendor changes an error code. The code is maybe 2,000 lines. Keeping it alive takes a meaningful fraction of someone's week.
This is the maintenance burden, and it's structurally underreported because it accumulates after the launch announcement. The demo cost is visible; the ownership cost is not. Google's classic paper on hidden technical debt in machine learning systems made the general point a decade ago: real ML systems are a small amount of model code surrounded by a large amount of glue, and that glue compounds silently (Hidden Technical Debt in Machine Learning Systems). LLM pipelines inherit every one of those problems and add a few of their own.
This article is an inventory of what you're signing up for, a way to estimate it, and the design choices that keep it from eating a team.
The maintenance surfaces, in order of how often they bite
1. The provider is not a stable dependency
Model providers deprecate snapshots, change default behavior, adjust rate limits, revise tokenizers, and retire endpoints. If you call a moving alias rather than a dated model, the underlying model can change without any action on your part. That is convenient until an evaluation set shows a regression you didn't cause and can't roll back by reverting your own code.
The maintenance work here is unglamorous:
- Pin to a specific model snapshot where the provider offers one, and treat an upgrade as a release with its own evaluation.
- Keep a fallback model configured, and test it. A fallback that has never been exercised is a comment, not a control.
- Read the provider's deprecation notices, or automate a check against their models endpoint and alert on the ones you use.
- Track tokenizer changes, because they change cost estimates and context limits even when the API looks identical.
None of this is hard individually. The problem is that it recurs on the provider's schedule, not yours, and it competes with product work for attention.
2. Prompts are code with worse tooling
A prompt is a program written in natural language with implicit inputs, implicit control flow, and no type system. It gets edited by people who aren't engineers, reviewed in a document rather than a diff, and versioned by filename if at all.
The result is predictable: prompt drift. Someone adds a sentence to fix one case, and three other cases change behavior. Nobody notices because there's no test. Two weeks later, a support ticket describes the regression in user language.
The minimum viable discipline is the same as for code: prompts in the repository, changes reviewed, a version identifier stored with every output, and a regression suite that runs on change. Whether you use a prompt management tool or a directory of YAML files matters less than the fact that the prompt has a version and an owner.
3. Evaluation sets rot
A golden set built at launch represents the traffic you imagined. Six months later, users have found new intents, the product has added a document type, and the set no longer covers the distribution. Worse, the expected outputs in the set were written against an older product policy, so some "failures" are actually correct new behavior.
Maintenance requires treating the evaluation set as a living artifact:
- Add real failure cases on a schedule, redacted, with corrected labels.
- Retire cases that no longer reflect policy, and record why.
- Track coverage by intent or document type, not just total count.
- Version the set alongside the prompt, so a score is always attributable to a specific pair.
This is the single most neglected maintenance task in LLM systems. It's also the one that determines whether you can safely change anything.
4. Input schemas drift upstream
Your pipeline expects a customer_id, a subject, and a body. The ticketing system adds a new required field, renames body to description, or starts sending HTML where it used to send plain text. The pipeline doesn't crash; it sends empty strings to the model and produces confident nonsense.
Schema drift is a data-contract problem, and the fix is a contract: validate at ingestion, fail loudly on violation, and alert the owner of the upstream system. A schema registry helps at scale, but a single validation function at the boundary plus an alert catches most of it.
5. Retrieval indexes go stale or wrong
If the pipeline includes retrieval, the index is a second system with its own failure modes: documents deleted upstream but still retrievable, chunks duplicated by a re-ingest, embeddings generated by a model version that no longer matches the query encoder, metadata filters that reference a field that was renamed.
Index maintenance is ongoing: incremental ingestion, deletion propagation, periodic re-embedding when the encoder changes, and a check that retrieval quality hasn't drifted. None of it is visible in the product until it produces a wrong answer.
6. Orchestration accumulates exceptions
Every incident adds a branch. "If the document is a scanned PDF, use the OCR path." "If the tenant is on the legacy plan, skip the reranker." "If the provider returns a 529, wait 30 seconds and retry once, but not for streaming requests." Six months in, the orchestration encodes a dozen undocumented decisions, and the only person who understands them has moved teams.
This is the pipeline-jungle problem from the technical debt literature: a data path that evolved organically, one special case at a time, until no one can reason about it end to end. The antidote is periodic consolidation — a scheduled quarter where you delete branches, merge paths, and write down the invariants — plus a rule that every new exception needs a test and a named owner.
7. Cost creeps in three directions
Token prices change, context sizes grow, and retry rates hide. A pipeline that cost $0.004 per item at launch can quietly reach $0.012 without any single decision that looks irresponsible. The drivers are usually:
- prompts that grew through accretion
- retrieved context that expanded because "more context helps"
- retries that double-charge on partial failures
- a model upgrade that costs more per token
- evaluation or logging jobs that call the model at scale
Cost maintenance needs an owner and a dashboard with a per-feature breakdown, not just a total. The total tells you something is wrong; the breakdown tells you where.
8. Observability decays
Logs are retained for 30 days, traces for 7, and the prompt version is stored in a field that a migration dropped. When an incident happens 45 days after the change, you can't reconstruct what the model saw. The pipeline technically has observability; it just doesn't have the observability you need at the moment you need it.
The maintenance task is to periodically test your own incident reconstruction: pick a random output from 60 days ago and try to explain it from stored data. If you can't, your retention or your trace schema is wrong.
9. Credentials and access rotate
API keys expire, service accounts get re-scoped, a provider adds a new permission model. Each rotation is a small operational event that can take the pipeline down if the key is baked into an environment variable that only one deploy path sets. Secret rotation is a solved problem in most orgs, but AI pipelines often sit outside the standard path because they were built by a data team with a separate deployment.
10. The ownership gap
The most expensive maintenance problem isn't technical. It's that nobody owns the pipeline after launch. It was built by an ML engineer who moved to a new project, operated by a data analyst who knows the symptoms but not the code, and reviewed by a platform team that doesn't know the model. Every change requires a meeting. Every incident requires archaeology.
A named owner, a runbook, and an on-call rotation are not bureaucracy. They're the difference between a two-hour fix and a two-week negotiation.
A worksheet for estimating the burden
You can't manage this without a number. The following is a rough model, not a forecast; substitute your own rates.
| Maintenance activity | Frequency | Effort per occurrence |
|---|---|---|
| Prompt change + regression run | 2–4 per month | 2–6 hours |
| Evaluation set refresh | Quarterly | 1–3 days |
| Provider deprecation / model upgrade | 1–3 per year | 3–10 days |
| Schema drift fix | 1–4 per year | 2–8 hours |
| Index re-embed or repair | 1–2 per year | 1–5 days |
| Cost investigation | Monthly | 2–4 hours |
| Incident response | 1–3 per quarter | 4–24 hours |
| Dependency and security updates | Monthly | 2–4 hours |
| Documentation and runbook upkeep | Continuous | 2–4 hours per month |
Add these up for a mid-sized pipeline and you get something in the range of 0.2 to 0.5 of an engineer, before any new feature work. That is the honest number to put next to the build estimate. If the feature's business value doesn't cover a fifth of an engineer plus infrastructure, a managed product or a simpler design is probably the better call.
The worksheet also exposes a useful asymmetry: the recurring items are mostly small, and the rare items are large. That means the risk isn't steady effort; it's the quarter where a provider deprecation, an index rebuild, and a schema change land in the same month. Staffing for the average underestimates the tail.
A month in the life
Abstract inventories are easy to dismiss. A composite month, drawn from common failure patterns rather than a single incident, shows how the work actually arrives.
Week one. The ticketing vendor ships an API update. body becomes description_html, and body is now empty for new tickets. The pipeline doesn't error; it processes a batch of empty bodies and emits "general inquiry" for 600 tickets. The regression is caught on Thursday when a support lead notices the routing queue is wrong. The fix is a mapping layer and a backfill, plus a validation rule that rejects empty bodies. Two days of work, most of it spent identifying which tickets were affected.
Week two. A prompt tweak to reduce over-classification is merged without a regression run, because the person making it didn't know the regression suite existed. It fixes the original complaint and shifts 4% of tickets from "billing" to "account," which nobody notices for eleven days. The eventual fix is to revert the prompt and re-apply the change with a proper evaluation. The cost isn't the revert; it's the eleven days of misrouted tickets and the trust lost by the routing team.
Week three. The provider announces that a model snapshot used by the summarization path will be retired in 60 days. The team has no recorded evaluation for that path, so the upgrade becomes an experiment: run both models on 200 recent documents, read the outputs, and argue about which is better. Three days of engineering time, plus a week of waiting for stakeholder review.
Week four. A finance review asks why inference spend rose 38% quarter over quarter. The investigation finds three causes: a context window that grew from 8k to 14k tokens after a retrieval change, a retry bug that re-sent successful batches after a timeout, and a new tenant with unusually long documents. Fixing the retry bug is an hour. Finding it takes a day and a half.
None of these events is dramatic. Together they consume most of a month's maintenance capacity, and none of them produced a new feature. That is the shape of the burden: not a single crisis, but a steady stream of small, context-heavy interruptions.
Why year three is worse than year one
Maintenance costs don't scale linearly with time, because each year adds constraints that make the next change harder.
The evaluation set grows, so regression runs take longer and cost more. The prompt accumulates exceptions, so each edit has more ways to interact badly. The number of consumers grows, so a behavior change that's fine for the original team breaks a downstream report. Documentation falls behind, so new engineers learn by reading code and asking questions. The original author leaves, taking the implicit knowledge of why certain branches exist.
This is ordinary software entropy, but AI pipelines feel it more sharply for two reasons. First, correctness is statistical, so you can't rely on a test suite to tell you whether a change is safe; you need labeled data and judgment. Second, the dependency you rely on most — the model — is outside your control and changes on someone else's schedule.
The practical implication is that the maintenance budget should grow, not stay flat. A team that allocates 20% of capacity in year one and 20% in year three will fall behind. The number that tends to work is closer to 30–40% once a pipeline is business-critical, and that capacity has to be protected from feature work rather than raided whenever a launch is behind.
What to write down
Documentation is the cheapest maintenance lever and the first thing sacrificed. For an AI pipeline, five documents carry most of the value.
The contract. Inputs, outputs, schemas, error codes, and the exact model and prompt versions in use. This is what a new engineer reads first and what an incident responder checks.
The invariants. What must always be true: every output has a source document, every write is idempotent, no tenant sees another tenant's data, every classification has a confidence state. Invariants are more useful than architecture diagrams because they tell you what to test.
The failure playbook. For each known failure mode: symptom, likely cause, first diagnostic step, mitigation, and who to call. "The nightly job processed zero rows" should have an entry. So should "the model started refusing benign requests."
The evaluation protocol. How to add a case, how to run the suite, what threshold blocks a release, and who signs off. Without this, evaluation becomes a ritual performed differently by each engineer.
The cost model. Cost per item, per tenant, and per feature, with the assumptions behind the token estimates. When someone proposes a change, this is how they estimate its impact.
These documents decay too, but they decay more slowly than memory, and they're the difference between a pipeline that survives a departure and one that has to be reverse-engineered.
Repair or rebuild
At some point the question shifts from "how do we maintain this?" to "should we replace it?" The signals that favor a rebuild are specific:
- The orchestration has more exception branches than happy-path steps.
- The evaluation set no longer reflects production, and rebuilding it would take longer than rebuilding the pipeline.
- The provider abstraction leaks everywhere, so switching models requires touching twenty files.
- Nobody can explain why a given prompt sentence exists.
- The cost of a change is dominated by fear rather than effort.
The signals that favor repair are equally specific: the domain logic is sound, the failure modes are understood, and the main problem is missing tests and documentation. Rebuilding a pipeline whose domain logic works usually reproduces the same logic with the same gaps, minus the institutional knowledge.
A middle path that often works: keep the domain logic, replace the plumbing. Extract the prompt-building, validation, and business rules into a library with tests, then rebuild the orchestration around it on a platform with better observability. This preserves what took the longest to learn and discards what accumulated accidentally.
What actually reduces the burden
Not all custom pipelines are equally expensive to own. A few architectural choices change the slope.
Put the provider behind a narrow adapter
Every call to a model should go through one module that owns retries, timeouts, request shape, response parsing, and error mapping. When the provider changes, you change one file. When you add a second provider, you implement one interface.
from dataclasses import dataclass
from typing import Protocol
@dataclass(frozen=True)
class Completion:
text: str
model: str
input_tokens: int
output_tokens: int
class LLM(Protocol):
def complete(self, system: str, user: str, *, max_tokens: int) -> Completion: ...
The interface is deliberately small. A large adapter that exposes every provider feature becomes a second product to maintain, and it leaks provider assumptions into callers. Keep the common path narrow and let genuinely provider-specific features live in a separate, explicitly named module.
Make the contract testable without the provider
A pipeline that can only be tested by calling a live model is a pipeline that gets tested rarely. Record real request/response pairs, store them as fixtures, and run the parsing, validation, and routing logic against the fixtures in CI. This won't catch model quality regressions, but it will catch the majority of code regressions, which are more common.
def test_extraction_handles_missing_total(fixture_loader):
raw = fixture_loader("invoice_missing_total.json")
parsed = parse_extraction(raw)
assert parsed.total is None
assert parsed.status == "needs_review"
Then keep a smaller, slower suite that calls the real model against the golden set on a schedule. Two tiers: fast contract tests on every commit, slow quality tests nightly or on prompt change.
Pin the model, version the prompt, store both
Every stored output should carry model_version, prompt_version, and schema_version. Without them, you can't answer "did this get worse because of the prompt, the model, or the input?" and you can't reproduce an incident. The storage cost is trivial; the debugging value is high.
Reduce the number of moving parts
Every additional component is a component that can fail, be deprecated, or need an upgrade. A pipeline with a queue, a worker, a vector database, a reranker, a cache, and an orchestration framework has six maintenance surfaces. Some are justified. Many are there because they were the default in a tutorial.
Ask of each component: what specifically breaks without it, and who will upgrade it in two years? If the answer to the second question is "nobody knows," that's a liability, not an architecture.
Build the kill switch before you need it
A feature flag that disables the AI path and falls back to manual processing is the cheapest incident mitigation you can build. It also changes the maintenance calculus: a bad model day becomes a degraded day rather than an outage. The flag should be per-feature, not global, and the fallback path should be exercised in staging.
Decide what not to build
The biggest maintenance saving is scope. A pipeline that classifies support tickets into eight categories is easier to maintain than one that also summarizes, routes, drafts replies, and extracts entities — not because the code is harder, but because each capability has its own evaluation set, prompt, failure modes, and stakeholder expectations. Ship one capability well, and add the second only when the first is boring to operate.
Build versus buy, without the religious war
Managed AI platforms and off-the-shelf pipelines reduce some maintenance and introduce others. The honest comparison is by maintenance surface, not by feature list.
| Surface | Custom pipeline | Managed platform |
|---|---|---|
| Model upgrades | Your responsibility | Vendor's, on their schedule |
| Prompt and eval tooling | Build or integrate | Usually included, often opinionated |
| Data residency | You control | Depends on vendor regions |
| Cost transparency | Full, with effort | Often per-seat or per-call markup |
| Differentiation | High | Limited to configuration |
| Incident debugging | You own the traces | Vendor owns part of the stack |
| Exit cost | Low if adapters are clean | Higher, especially with proprietary eval data |
| Time to first version | Days to weeks | Hours to days |
The pattern I'd recommend: buy the commodity layer (model access, tracing, evaluation harness, vector storage) and build the part that encodes your domain. The domain logic — how your documents are structured, what "correct" means in your business, which exceptions matter — is where custom work earns its keep. The plumbing is where it becomes a tax.
There's a caveat: managed platforms can hide failures in ways that make debugging harder, and their abstractions can be hard to escape. If your feature is core to the product and you expect to iterate on it for years, owning the orchestration and the evaluation data is worth the cost. If it's a supporting capability, buying is usually cheaper.
The two-year question
A useful forcing function before starting a custom pipeline: describe the system as you expect it to exist in two years, and ask who will perform each of these tasks.
- Upgrade the model when the current one is deprecated.
- Re-run the evaluation set after a prompt change.
- Investigate a cost spike.
- Fix a schema change from an upstream system.
- Re-embed the corpus after an encoder change.
- Answer an auditor's question about a specific output.
- Decide whether to adopt a new provider capability.
If the answer for most of them is "the team that built it," and that team is one person with other priorities, the pipeline will decay. That's not a reason to avoid building; it's a reason to build less, document more, and choose components with long support horizons.
The maintenance burden of a custom AI pipeline is real, but it isn't mysterious. It's a list of recurring tasks with known frequencies, and it can be estimated, staffed, and reduced. The teams that do well are not the ones with the cleverest architecture. They're the ones that treated the pipeline as a product with an owner, a budget, and a lifecycle — rather than a demo that happened to reach production.