A patch lands that adds a 90-line function to app/services/invoice_totals.py. It handles currency rounding, applies a discount, and formats the result. It has type hints, a docstring, three tests, and it matches the surrounding code closely enough that a reviewer can't tell it wasn't written by the person who wrote the rest of the file.
There's a money.py in the same package, written two years ago, with apply_discount and round_currency in it, handling the same rounding rules that took a support escalation and a tax audit to get right. The new function reimplements both, slightly differently. It's the difference that matters: the old one rounds half-up on the tax-inclusive subtotal, the new one rounds half-even on the pre-tax amount. Both are defensible. Only one matches what finance reconciles against.
The patch passed CI. It's plausible. And plausibility is exactly the property you can't review for, because it's the property that makes review feel unnecessary.
Plausibility is a style property, not a correctness property
Code generation models are trained to produce code that looks like the code around it. That's genuinely useful — a patch that follows local conventions is easier to read and less likely to introduce a second way of doing things. But it also means the strongest signal a reviewer has ("this looks like it belongs here") is produced by the same process that produced the code, and is therefore not evidence about anything.
Compare the failure signatures:
| Human patch | Generated patch | |
|---|---|---|
| Typical failure | Incomplete, inconsistent, visibly unsure | Complete, consistent, confidently wrong in one local decision |
| Signal at review | Naming drift, missing tests, TODO comments | Clean diff, tests pass, described as done |
| Discovered by | Reading carefully | Production, or a year later |
| Cost curve | Front-loaded | Back-loaded |
Front-loaded cost is annoying but cheap: you catch it before merge. Back-loaded cost is the expensive kind, because it arrives after the code has dependents, after the author has moved on, and after the tests have calcified around the current behavior.
That asymmetry is the entire economic argument, and it's usually missing from discussions that compare "time to write" against "time to review." Writing time is the smallest term.
The five places the cost actually lands
Vague claims about "maintenance burden" aren't actionable. Here's where the money goes, in rough order of size for a typical application codebase.
1. Review minutes, at a volume that scales. A 400-line generated diff takes 25–40 minutes to review properly and 4 minutes to skim. Skimming is what happens when the queue is full, so the practical cost isn't reviewer time — it's the fraction of merges that received real review. If generation triples PR volume and review capacity is fixed, review depth falls by roughly two-thirds. Nothing announces this. Approval timestamps just get shorter.
2. Duplication. Generated patches don't know what's three directories away unless it's in the context. They reimplement. Every reimplementation is a second place to fix a bug, and the two copies diverge the first time only one of them gets the fix.
3. Wrong abstraction. Given one example, a model will often generalize: a helper for one caller, a base class for one subclass, a config flag for one tenant. Each of these is a commitment that costs more to undo than it cost to write.
4. Tests that encode the implementation. A test written alongside the code tends to assert what the code does rather than what it should do. That's fine until the implementation is wrong, at which point the test defends the bug.
5. Knowledge debt. Nobody on the team has a mental model of code they didn't design. That's tolerable when the code is simple. It's expensive when the code is the kind of thing that requires a model of the domain to debug at 3am.
None of these appear in a per-token cost calculation, and all of them show up in the only number that matters: how long it takes to change the system safely.
Duplication is measurable, so measure it
The good news about duplication is that you don't have to argue about it. Run a detector, look at the report, and see whether generated patches are making it worse.
npx jscpd --min-tokens 50 --reporters console,json \
--output ./tmp/jscpd --ignore "**/node_modules/**,**/*.test.*,**/migrations/**" .
For Python, pylint --disable=all --enable=duplicate-code or pmd's CPD both work. The exact tool matters less than running it on a schedule and keeping the number visible.
Two things to watch beyond the headline percentage:
New clusters, not total duplication. A codebase can sit at 4% duplication forever because of generated DTOs and migration boilerplate. What matters is whether new near-duplicates appear in hand-written application code, and whether they pair a generated function with a pre-existing one.
Divergence inside a cluster. Two copies that are textually identical are a maintenance annoyance. Two copies that differ in one line — the rounding mode, the retry count, the timezone assumption — are a latent incident. A diff-aware check helps:
# Find helpers defined more than once with similar names
git grep -n -E "def (apply_discount|round_currency|calculate_total|compute_tax)" \
-- '*.py' | sed 's/:.*def /: /'
If that returns three definitions across two modules, the review question for the next patch touching money is not "is this function correct" but "which of these three is canonical, and why are there three?"
💡 Practical note: put the canonical helper's path in a
CONTRIBUTINGnote or, better, in the agent's context. Most duplication from generated patches is a context problem, not a judgment problem — the model couldn't seemoney.py, so it wrote its own.
The wrong-abstraction tax is real and it compounds
Given a task like "add PDF export to the reports page," a generated patch will frequently produce something like:
class BaseExporter(ABC):
@abstractmethod
def fetch(self, report_id: str) -> dict: ...
@abstractmethod
def render(self, data: dict) -> bytes: ...
class PdfExporter(BaseExporter):
def fetch(self, report_id: str) -> dict:
return reports.get(report_id)
def render(self, data: dict) -> bytes:
return pdf.render(data)
Two classes, an ABC, and an indirection layer for a single caller. It's not wrong — it's premature. And it becomes load-bearing the moment someone writes isinstance(exporter, BaseExporter) or adds a second exporter that half-fits the interface.
The cost isn't the 20 lines. It's that the next person who needs a CSV export now has to fit the shape, and the shape was designed with one example in view. Undoing it means touching every caller, which is why it doesn't get undone.
A useful review heuristic: for any new abstraction in a generated patch, ask how many callers exist today. If the answer is one, ask for the concrete version. If the answer is one but "there will be more," ask which specific second use case is committed, and when. "There will be more" is the most expensive sentence in software.
This is also a place where a prompt can help before review:
Prefer the smallest change that solves the task. Do not introduce base classes,
interfaces, or configuration flags for a single call site. If you believe an
abstraction is warranted, describe the second caller that requires it instead
of writing the abstraction.
That one paragraph eliminates a meaningful fraction of the diffs you'd otherwise reject — which is cheaper for everyone than rejecting them.
Tests written from the implementation defend the implementation
The subtle version of "tests pass" is that the tests were generated from the same understanding that produced the bug. When the implementation rounds half-even and the test asserts half-even, the suite is green and wrong.
Three checks catch most of it.
Does the test name describe behavior or implementation? test_rounds_currency_half_even is a behavior claim you can check against the tax rules. test_calls_round_with_mode_half_even is a restatement of the code.
Is there a case the implementation would fail if the requirement changed? A test suite for a discount calculator that never includes a discount of exactly 100% or a zero-amount invoice isn't testing the requirement; it's testing the code path the author happened to think of.
Would this test fail if someone broke the feature in an obvious way? Mutate one line by hand and run the suite. If nothing goes red, the test is decoration. This takes two minutes per PR and it's the single most informative review action available.
# Cheap mutation check, run by hand during review
# 1. Break the implementation on purpose
# return round(x, 2) -> return round(x, 3)
# 2. Run the tests
# expect: at least one failure
# 3. Revert
For generated tests specifically, add one more: do the tests cover the error path the PR claims to fix? A patch titled "handle expired tokens gracefully" with tests only for valid and malformed tokens hasn't tested the change.
The capacity ceiling nobody budgets for
Here's the arithmetic that matters more than any individual diff.
If a team of five reviews 60 PRs a week at 20 minutes each, that's 20 hours — a fifth of a person, sustainable. Double PR volume without changing anything else and review consumes 40 hours a week. Either review depth halves or the queue grows. Most teams choose depth, silently, because the approval button is right there.
The consequence is a strange one: generation gets cheaper and merging gets more expensive, and the second curve is the one that determines throughput. A team that doubles output and halves review depth hasn't doubled anything; it's moved the cost to the incident.
The mitigations that actually work are structural, not disciplinary:
- A diff-size budget enforced in CI. Cap files and lines per PR. Large changes must arrive as a stack of small ones, which are reviewable in order and revertable individually.
- Mechanical checks promoted to gates. Lint, type check, undefined-name detection, test-delta inspection. Anything a script can decide shouldn't consume a human's attention.
- A second reviewer only for risk categories. Auth, billing, migrations, public contracts. Everything else gets one careful reviewer instead of two distracted ones.
- Explicitly permitted low-review categories. Test scaffolding, codemods, dependency bumps with a green suite. Naming these frees real capacity for the diffs that need it.
The point isn't to slow generation down. It's to stop pretending review is free.
Churn tells you whether the code was worth writing
Lines generated is a vanity metric. Lines that survive is closer to the truth, and lines that get rewritten within 90 days is the metric that correlates with "we wrote this too fast to be right."
# Files with the most rewrites in the last 90 days, from git history
git log --since="90 days ago" --numstat --pretty=format: -- '*.py' \
| awk 'NF==3 && $1 != "-" {add[$3]+=$1; del[$3]+=$2}
END {for (f in add) if (add[f]+del[f] > 0)
printf "%6d %6d %s\n", add[f], del[f], f}' \
| sort -rn | head -30
A file that accumulates both high additions and high deletions is churning: written, rewritten, rewritten again. High churn in files that recently received generated patches is worth a look, and the pattern is usually one of two things — an abstraction that didn't fit, or a misunderstanding of the domain that took three attempts to correct.
Churn analysis has an obvious trap: files that are genuinely volatile (a router, a schema, a config surface) will always look bad. Compare churn before and after generation started landing in a module, rather than across modules.
What a plausible diff costs at 3am
The cost that never makes it into any spreadsheet is the one that appears during an incident: nobody has a model of the code that's failing.
A person who wrote a function knows why the guard clause is there, which invariant it protects, and what happens if you remove it. A generated function has that reasoning only if someone wrote it down and someone else read it. Most don't. The docstring says "validates the payload." It doesn't say "rejects partial addresses because the tax service returns a 500 for them, and the retry loop turns that into a duplicate charge."
So during an incident, the on-call engineer reads the code cold, at the worst possible time, with the least possible patience. They can see what it does. They can't see what it's protecting against. The fastest path is usually to add a guard that makes the symptom disappear, which is how a second latent bug gets created.
The mitigation is unglamorous and cheap: require a non-obvious-decision comment on generated code that encodes a domain rule. Not "what does this do" — the code says that. "Why is this number 4 and not 3," "why is this retried here and not upstream," "which upstream behavior makes this necessary." If the model can't state a reason, that's information: the decision may be arbitrary.
The schema blind spot
Schema and migration code deserves its own warning, because it combines every risk at once: the code is plausible, the tests rarely exercise it at scale, and reverting it is often impossible.
A generated migration that adds an index will look correct. What it won't encode is whether the table has 40 million rows, whether the index can be built without blocking writes, and whether the query planner will actually use it. On a small table, all three are moot. On a large one, the difference between CREATE INDEX and CREATE INDEX CONCURRENTLY is a maintenance window.
The same applies to column changes. A generated patch that adds NOT NULL with a default, or tightens a constraint, may be logically right and operationally wrong on a table that's still receiving writes from a previous deploy. Those are the migrations that take the site down for eleven minutes at an inconvenient time.
The practical guard is a rule, not a review technique: generated migrations are drafts. Every one goes through a person who can state the table's approximate size, the locking behavior of the operation, and the rollback path. If they can't, the migration doesn't ship. That's a two-minute conversation that prevents a category of outage no amount of review attention reliably catches, because the failure isn't visible in the SQL.
🚨 Production warning: a generated migration that drops or renames a column will pass every test in a suite that builds its schema from migrations. Tests create the final schema; they never run against the old one. The only reliable check is a staging deploy against a recent production-shaped snapshot.
Reverting is the cheapest control you have, and it's underused
When a plausible diff turns out to be wrong, the cheapest correction is a revert. It's also the option teams reach for last, because by the time the wrongness is visible, the patch has accumulated follow-on work that depends on it.
That dependency is preventable. Two habits make reverts cheap:
Keep generated patches small enough to revert without a merge conflict cascade. This is the same diff budget argument, from a different direction. A 600-line patch touching 20 files can't be cleanly reverted after a week. A 90-line patch touching three files usually can.
Don't let generated patches become load-bearing before they've been exercised. The risky window for a plausible diff is the period between merge and first real traffic. Code that sits in main for three weeks gathering dependents before it runs in production is code you can no longer cheaply undo. Ship it behind a flag, exercise it, then build on it.
A revert-based workflow also changes review incentives in a useful way. If the default response to a subtle wrongness is "revert, then fix forward deliberately," reviewers stop feeling obligated to catch everything before merge — an impossible standard that produces performative scrutiny rather than real scrutiny. The gate becomes "is this safe to try," which is a question people can actually answer.
Where generation is genuinely cheap
It would be dishonest to end up arguing that generated code is always a bad trade. It isn't. The economics are strongly positive in specific shapes:
- Mechanical breadth. Renaming a parameter across 60 call sites, migrating a test file to a new assertion library, adding a field to 30 DTOs. High line count, near-zero decision content, trivially reviewable by sampling.
- Test scaffolding for behavior you've already defined. Given a table of input/expected pairs that a human wrote, generating the test file is pure win, because the decisions are already made.
- Throwaway analysis. A script to answer one question about production data, run once, deleted. The maintenance cost is zero because nothing is maintained.
- Greenfield prototypes. Code whose entire purpose is to be evaluated and rewritten. Plausibility is a feature here.
The pattern: generation is cheap where the decision content is low, and expensive where the decision content is hidden inside plausible-looking code. The failure mode is applying the first category's economics to the second.
Review for what the diff doesn't contain
Most review attention goes to lines that exist. With generated patches, the more informative signal is often the absence.
A patch that adds a new external call and doesn't add a timeout. A patch that adds a write path and doesn't add an idempotency key. A patch that adds a retry loop and doesn't add jitter. A patch that adds a query inside a loop and doesn't add a limit. A patch that touches a model and doesn't add a migration. None of these appear as changed lines, which is precisely why they survive review: there's nothing to point at.
The technique that works is to name the expected companions before reading. For a patch that makes an outbound HTTP call, the expected companions are: timeout, retry policy, error classification, and a log line with the target and duration. Then check each one. You're not reading for defects; you're checking a short list against the diff, which takes a minute and catches the omissions that no amount of line-by-line reading would.
This is also where generated code differs most from human code. A human adding a call to a flaky third-party API usually remembers the timeout, because they remember the last incident. The model has no incident memory. It has the shape of the call, and the shape of the call rarely includes a timeout.
A cost model you can actually compute
If you want to argue about this with numbers instead of adjectives, here's a small model. It won't be precise; it will be more precise than "it feels faster."
cost_per_merged_line =
(review_minutes * loaded_hourly_rate / 60
+ expected_rework_lines * minutes_per_rework_line * rate / 60
+ expected_incident_share * incident_cost)
/ merged_lines
The terms you can measure:
review_minutes— approval timestamp minus first review request, minus queue time. Imperfect but directionally right.expected_rework_lines— lines changed within 90 days of merge, fromgit logon the merge commit's files.expected_incident_share— incidents attributed to recently merged code, divided by recent merges. Small numbers, so use quarters not weeks.incident_cost— engineer-hours to resolve, times rate, plus whatever customer impact you can quantify.
The point isn't the fourth significant digit. It's that expected_rework_lines and expected_incident_share are the terms people leave out, and they're the ones that decide whether a generation workflow is actually cheaper. A team whose generated patches are merged 2× faster and rewritten 3× more often has gotten slower, and the model makes that visible.
What I'd actually do
Three practices, in order of return.
First, constrain the context before you constrain the output. Most duplication and wrong-abstraction problems are context failures. Give the agent the canonical helpers, the domain rules, and the local conventions, and ask it to reuse rather than reimplement. This is cheaper than review and it fixes the cause.
PROMPT_PREAMBLE = """
Before writing code, list the existing modules in this package and the helpers
they already provide. Reuse them. If you need behavior that does not exist,
say so in one sentence before implementing it.
Do not add an abstraction for a single caller. Do not add a config flag unless
a second caller exists today.
If you make a decision that a future reader could not infer from the code
(a magic number, an ordering, a retry count), leave a one-line comment
explaining what would break without it.
"""
Second, budget the diff. Hard limits on files and lines per PR, enforced in CI. Not to prevent large changes, but to force them into reviewable units. This is the single highest-leverage process change, because it fixes the review-depth collapse directly.
Third, own what you merge. A generated patch with a human approver is a human patch. That framing sounds harsh and it's the only one that produces the right behavior: if you approved it, you're the person who knows why the guard clause exists, and if you don't know, you're not done reviewing.
The alternative — treating generated code as someone else's output that you merely shepherded into main — is how a codebase becomes something nobody understands while everyone keeps shipping. Plausible diffs are cheap to produce and expensive to own, and the only real defense is refusing to own things you haven't read.