The docs said the endpoint accepted a limit query parameter. The code had renamed it to page_size two sprints ago. The integration test passed, because the test called the function directly and never touched the HTTP layer. The documentation page passed review, because reviewers read prose, not handler signatures. And the only person who noticed was a customer, six days later, after building an integration that returned 400s on every request.
That's the shape of documentation drift. It's not a dramatic failure. It's a quiet divergence between what a repository claims and what it does—and it compounds. Every stale example, every renamed flag, every “configurable” value that's actually hardcoded makes the repository less trustworthy for the next person, the next agent, and the next integration. In 2026, when AI coding agents read your docs and implement against them, a stale doc isn't just unhelpful. It's an active source of wrong code.
The tooling landscape has split into two camps: deterministic anchoring that catches structural drift with zero false positives, and LLM-based semantic analysis that catches meaning-level drift but generates noise. Neither is sufficient alone. This is how to combine them into a pipeline that actually holds.
📋 Table of Contents
- Why Docs Drift Is a Trust Problem, Not a Writing Problem
- 1. Anchor Docs to Stable IDs, Not Line Numbers
- 2. The Pair Ledger: Fingerprint the Last Confirmed Match
- 3. Anchor Liveness: Every Backtick Is a Verifiable Claim
- 4. Coverage Gates: Silence Is Not Compliance
- 5. Extract Claims With One Agent, Verify With Another
- 6. Generate Tests From Docs and Execute Them
- 7. Local Categorization and External Filtering for Semantic Drift
- 8. Treat OpenAPI Specs as a Separate Drift Surface
- 9. Wire It Into CI Without Drowning in False Positives
- What I'd Build, What I'd Skip, and the Decision That Matters
Why Docs Drift Is a Trust Problem, Not a Writing Problem
The obvious fix for stale documentation is “write better docs” or “update the docs when you change the code.” Both fail for the same reason: the feedback loop is too slow and too social. A developer changes a function signature, CI passes, the PR merges, and the documentation page three directories away silently becomes a lie. Nobody is alerted because nobody is watching the connection between the two.
The research on this is unambiguous. CASCADE, a 2026 tool for detecting code-documentation inconsistencies, notes that even minor mismatches “can confuse API users, introduce new bugs, and increase overall maintenance effort” . The problem isn't that developers don't care. It's that the connection between a doc sentence and the code it describes is usually implicit, and implicit connections break silently.
LyingDocs frames the same issue in terms of trust: “Modern repositories are read by more than humans. They are read by teammates, new contributors, users, reviewers, downstream integrators—and increasingly by AI agents. That only works if the repository can be trusted” . A repository that says one thing and does another is not just inconvenient. It's unreliable as a source of truth for anyone building on top of it.
The tools that address this fall into three architectural families, each with a different precision/recall trade-off:
| Approach | Detection method | Strength | Weakness |
|---|---|---|---|
| Deterministic anchoring | Git hashes, stable IDs, grep-based anchor checks | Zero false positives; fast; language-agnostic | Blind to semantic drift when identifiers still exist |
| LLM semantic analysis | Claim extraction, code investigation, or test generation | Catches meaning-level mismatches | Noisy; requires filtering and human confirmation |
| Contract/schema diffing | OpenAPI/GraphQL schema comparison, live response diffing | Catches API surface drift precisely | Limited to APIs with formal contracts |
The mistake most teams make is picking one family and expecting it to cover everything. A deterministic tool will never catch a docstring that says “returns the user's email” when the code returns the user's username—every identifier still exists. An LLM tool will drown you in false positives if you run it on every commit without deterministic pre-filtering. And an OpenAPI diff tool won't catch a README that describes an endpoint that was never implemented.
The rest of this article is a pipeline that layers these approaches in order, starting with the cheapest and most reliable signals and escalating to semantic analysis only where it's needed.
🔍 Why this matters: A false positive in a drift detector isn't just annoying. It trains developers to ignore the tool. CASCADE's authors make this point directly: “Since automatic reports still require human confirmation, false positives carry serious consequences: wasting developer time and discouraging practical adoption” . Precision is not a nice-to-have. It's the difference between a gate that holds and a gate that gets disabled.
1. Anchor Docs to Stable IDs, Not Line Numbers
Scenario: A design doc links to src/auth/handler.ts#L42-L58 to illustrate the token-refresh sequence. A refactor moves the function to src/auth/token/handler.ts and shifts every line by 30. The link now points to an unrelated section of the file. The doc still looks correct in review, because the prose is unchanged.
Why line-number links fail: Line numbers are positional, not semantic. They break on every refactor, every formatter run, every file move. A doc that depends on them is coupled to the exact layout of the code at the moment it was written.
Solution: Mark the code block once with a stable identifier, and cite that identifier from prose. The identifier travels with the block through moves, renames, and reformatting. docsync implements this with inline directives: a comment in the source file declares the block, and a link in the doc cites it by ID.
// ds:def id=sess-save-k7m2p4xq owner=@auth stability=api
func (s *Store) SaveSession(ctx context.Context, sess Session) error {
// ...
}
Every write goes through [`SaveSession`](ds:block?id=sess-save-k7m2p4xq).
The id is a short opaque string with a human-readable label prefix. The suffix is the identity; the label is just a hint . When the function moves to a different file, the directive moves with it, and the doc link still resolves. When the function is deleted, ds check reports the citation as broken, because the block no longer exists.
docsync also renders code snippets and values from live source at build time, so the doc never holds a stale copy . That's a stronger guarantee than “the link is correct.” It's “the content is generated from the thing it describes.”
Why this works: The indirection between doc and code is explicit and machine-readable. The tool doesn't have to guess which code a sentence refers to, because the author declared it. That declaration is the foundation for every downstream check—impact analysis, staleness reporting, and CI gating.
💡 Practical note: Stable IDs are only stable if they're never reused. If you delete a block and later create a new block with the same ID, old citations will silently point to the wrong code. Use a random suffix, not a sequential counter, and never recycle IDs.
2. The Pair Ledger: Fingerprint the Last Confirmed Match
Scenario: A doc and its corresponding source directory are both changed in the same sprint. The doc is updated to describe a new configuration option, but the code change actually removed a validation step that the doc still implies. Both files have new commit hashes. Nothing looks broken, because the doc and code moved together.
Why commit-level diffing is insufficient: A diff tells you that something changed, not whether the change is coherent. Two files can change in the same commit and still contradict each other.
Solution: Record a fingerprint of both sides when a human confirms they match, and flag any one-sided change mechanically. staledocs calls this a pair ledger: every doc/code pair stores the git blob hashes from the last acknowledged coherent state. From then on, if the code changes and the doc doesn't—or the doc changes and the code doesn't—the pair is flagged.
The detection is graded, not binary. If the code moved but the doc didn't, the severity depends on whether the change touches something the doc actually names. Unrelated churn inside a wide pair downgrades to amber; a change to a named identifier or path stays red . That grading is what keeps the signal actionable.
# .staledocs.yaml — pair declarations
version: 1
gate: warn
source:
include: ["src/**"]
docs:
include: ["docs/**/*.md", "README.md"]
pairs:
- doc: docs/auth.md
code: ["src/auth/**"]
- doc: docs/token.md
code: ["src/auth/token_*.py"]
The ledger lives in .staledocs/pairs/ as one JSON file per pair and is meant to be committed. A merge conflict in a ledger entry fails safe: the entry stops parsing, the pair reverts to unacked, and gets re-checked .
Why this works: The fingerprint is a deterministic record of a human judgment. It doesn't try to infer coherence from the diff. It anchors the last known-good state and reports divergence from it. That's a much weaker claim than “these docs are correct,” but it's a claim the tool can make with perfect reliability. And a reliable weak signal is more useful than an unreliable strong one.
⚠️ Gotcha: The pair ledger is only as good as the pairs. If a doc is paired with a source directory that's too broad, unrelated changes will constantly trigger amber warnings, and developers will learn to ignore them. Pair tightly—a doc to the specific module it describes—and use the coverage gate to catch docs that aren't paired at all.
3. Anchor Liveness: Every Backtick Is a Verifiable Claim
Scenario: A README says “run npm run build:prod -- --minify to produce a minified bundle.” The build:prod script was removed two releases ago. Nobody notices until a new contributor follows the instructions and gets a “missing script” error.
Why prose review misses this: A human reviewer reads the sentence and checks whether it makes sense. They don't grep the package.json for the script name. The claim is embedded in a backtick, which looks like code but isn't executed or validated by anything.
Solution: Extract every backtick-enclosed token from the documentation and verify that it still exists on the paired code side. staledocs calls this anchor liveness: docs naturally quote identifiers, CLI flags, and paths in backticks, and each one is a claim that can be mechanically checked. The code is only grepped, never parsed, which keeps the tool language-agnostic .
import re
from pathlib import Path
def extract_anchors(doc_text: str, min_length: int = 3) -> set[str]:
"""Pull backtick-enclosed tokens that look like identifiers or paths."""
anchors = set()
for match in re.finditer(r'`([^`\n]+)`', doc_text):
token = match.group(1).strip()
if len(token) >= min_length and re.search(r'[A-Za-z_./-]', token):
anchors.add(token)
return anchors
def check_anchors(doc_path: Path, code_roots: list[Path]) -> list[dict]:
anchors = extract_anchors(doc_path.read_text())
findings = []
for anchor in anchors:
found = any(
anchor in path.read_text(errors="ignore")
for root in code_roots
for path in root.rglob("*")
if path.is_file()
)
if not found:
findings.append({"doc": str(doc_path), "anchor": anchor, "status": "dead"})
return findings
The report names the exact doc line that rotted, not just the file. That matters for fixing it: “docs/setup.md line 34 references build:prod, which no longer exists” is actionable. “Something in setup.md is stale” is not.
Why this works: Backtick anchors are a proxy for the doc's factual claims about identifiers. They're not a complete representation of meaning, but they cover a large class of high-frequency drift: renamed functions, removed CLI flags, moved files, deleted config keys. And because the check is grep-based, it works identically across languages and monorepos.
🧠 The important part: Anchor liveness catches the drift that's invisible to prose review because the prose still reads correctly. The sentence “run
build:prod” is grammatically fine and semantically plausible. Only a mechanical check against the actual scripts can tell you it's a lie.
4. Coverage Gates: Silence Is Not Compliance
Scenario: A new module is added to src/ with no corresponding documentation. The drift detector reports zero issues, because there's no pair to check and no anchor to verify. The repository looks clean. The new module is completely undocumented.
Why “no findings” is a misleading signal: A detector that only checks existing pairs and existing anchors will always report success on undocumented code. Silence looks like compliance, but it's actually a coverage gap.
Solution: Add a coverage gate. Every source file must belong to at least one doc pair, and every doc must be classified as paired, standalone, or global. New files with no owner show up immediately. staledocs makes this explicit: “Silence is never coverage” .
def coverage_gaps(source_files: set[str], paired_globs: list[str],
standalone: set[str], docs: set[str]) -> dict:
unpaired = {
f for f in source_files
if not any(fnmatch.fnmatch(f, g) for g in paired_globs)
}
unclassified_docs = {
d for d in docs
if d not in standalone and not any(d in p for p in paired_globs)
}
return {"unpaired_source": sorted(unpaired),
"unclassified_docs": sorted(unclassified_docs)}
The gate should start in warn mode on a brownfield repo. The first run will print dozens of findings—that's the point. Triage them into real gaps, intentionally standalone docs, and scope adjustments, then flip to strict once the backlog is cleared.
Why this works: Coverage gates turn documentation from an opt-in activity into a bounded obligation. You don't have to document everything perfectly, but you do have to decide, explicitly, what is documented and what isn't. That decision is recorded in the config, reviewed in the PR, and enforced in CI.
🚨 Production warning: Don't turn on
strictmode before triaging the initial findings. A gate that fails on day one for a hundred pre-existing gaps will be disabled by lunch. Start inwarn, clear the backlog, then enforce.
5. Extract Claims With One Agent, Verify With Another
Scenario: A docstring says “returns the user's email address.” The function actually returns the username. Every identifier in the docstring still exists. Anchor liveness passes. Pair ledger passes. The only way to catch this is to read the prose and compare it to the code's behavior.
Why deterministic checks can't catch semantic drift: Identifiers are a proxy for meaning, not meaning itself. When the prose misdescribes behavior but the tokens are intact, grep-based checks have nothing to latch onto.
Solution: Use an LLM to extract claims from the documentation, then use a separate agent to investigate whether the code actually implements those claims. LyingDocs implements this as two agents: Hermes reads the docs and turns claims into targeted investigation tasks; Argus executes those tasks against the real codebase using a configurable backend (OpenAI Codex, Claude Code, or a local agent loop) .
def audit_claims(doc_path: str, code_root: str, agent_backend: str):
# Phase 1: extract claims and plan an audit.
claims = hermes_extract_claims(doc_path) # e.g. ["returns email", "paginates by 50"]
# Phase 2: investigate each claim against the actual code.
findings = []
for claim in claims:
result = argus_investigate(claim, code_root, backend=agent_backend)
if result["status"] == "contradicted":
findings.append({
"doc": doc_path,
"claim": claim,
"evidence": result["code_reference"],
"explanation": result["explanation"],
})
return findings
The separation matters. Hermes doesn't need to understand the codebase; it needs to read the docs carefully and identify what they're claiming. Argus doesn't need to read the whole doc; it needs to answer specific questions about the code. Each agent has a narrower task and a clearer failure mode.
Why this works: Claim extraction turns a vague question (“is this doc accurate?”) into a set of falsifiable questions (“does the code return an email?”). The verifier agent can then investigate each question independently, which makes the output traceable and the false-positive rate manageable.
💡 Practical note: Run semantic audits on a schedule, not on every commit. They're slower and more expensive than deterministic checks, and most semantic claims don't change between commits. A nightly run or a pre-release audit catches the drift without adding latency to the PR loop.
6. Generate Tests From Docs and Execute Them
Scenario: A doc says “calling parse_config with an empty string raises ValueError.” The code now returns None instead. The docstring is semantically wrong, but it's still plausible. No static check will catch it, because ValueError and None are both valid identifiers that exist in the codebase.
Why static analysis stops here: The claim is about behavior under execution, not about the presence of a symbol. To verify it, you need to execute the code.
Solution: Use an LLM to generate tests directly from the documentation, then run those tests against the actual code. CASCADE does exactly this: it generates unit tests from natural-language docs and treats any test failure as a potential inconsistency. To reduce false positives, it also generates code from the docs and cross-checks: an inconsistency is reported only when the existing code fails a test that the generated code passes .
# Conceptual flow for doc-derived test generation.
def cascade_check(docstring: str, function_source: str, test_runner):
# 1. Generate tests from the documentation alone.
tests_from_doc = llm_generate_tests(docstring)
# 2. Run the tests against the existing implementation.
existing_results = test_runner(tests_from_doc, target=function_source)
# 3. Generate an implementation from the documentation.
generated_impl = llm_generate_implementation(docstring)
# 4. Run the same tests against the generated implementation.
generated_results = test_runner(tests_from_doc, target=generated_impl)
# 5. Report a mismatch only when the existing code fails
# and the doc-derived implementation passes.
if existing_results.failed and generated_results.passed:
return {"status": "inconsistent", "evidence": existing_results.errors}
return {"status": "consistent"}
CASCADE's evaluation on 71 inconsistent and 814 consistent Java pairs, plus additional Java, C#, and Rust repositories, found 13 previously unknown inconsistencies, 10 of which were fixed . The cross-check design is the key: it filters out tests that are wrong because the documentation is ambiguous or the LLM misunderstood it.
Why this works: Execution-based verification is the strongest signal available for behavioral claims. It doesn't rely on identifier matching or semantic similarity. It asks the code to do what the doc says it does, and reports when it doesn't.
⚠️ Gotcha: This approach requires the documentation to be specific enough to generate tests from. A doc that says “handles errors gracefully” produces no useful test. Start with well-structured docstrings and API references, where the claims are concrete and testable.
7. Local Categorization and External Filtering for Semantic Drift
Scenario: You run an LLM-based drift detector across a large repository. It flags 400 potential inconsistencies in the first run. Most are false positives: the model misread a comment, flagged an intentional simplification, or reported an “under-promise” (code that does more than the doc says) as a problem. The signal is buried.
Why naive LLM prompting produces noise: A single prompt that asks “does this code match this doc?” conflates three different questions: is the code missing something the doc promises, does the code contradict the doc, and does the code do something the doc doesn't mention. The model conflates them too.
Solution: Structure the LLM's task into explicit categories, and filter out the categories you don't care about. DocPrism does this with local categorization and external filtering. The prompt asks the model to answer three paired questions: is any core documentation not implemented, does the code conflict with the docs, and is any code undocumented. Each answer gates a follow-up that lists the specific snippets. Then external filtering drops the “under-promise” category, because undocumented code is a coverage problem, not a drift problem .
# The three paired questions DocPrism asks (simplified).
CHECK_KEYS = [
"is_documented_behavior_missing_from_code", # over-promise
"does_code_conflict_with_documentation", # direct mismatch
"is_code_undocumented", # under-promise
]
def docprism_prompt(docstring: str, code: str) -> str:
return f"""Answer each question with Yes or No, then list evidence.
1. Is any core part of the documentation not implemented in the code?
2. Does the code correctly implement what is mentioned in the documentation?
3. Is some code not documented or mentioned in the documentation?
[DOCUMENTATION]
{docstring}
[/DOCUMENTATION]
[CODE]
{code}
[/CODE]
"""
DocPrism uses LLaMA 3.1-70B with temperature set to 0 for deterministic output. On its Python evaluation set, it reported function-level precision of 0.64 and accuracy of 0.92, with a flag rate of 11% . Those numbers are not high enough to auto-fix without review. They are high enough to prioritize which docs a human should look at first.
Why this works: Categorization makes the model's reasoning inspectable. You can see whether a finding is an over-promise, a direct contradiction, or an under-promise, and you can filter at the category level. That's much more useful than a binary “drift / no drift” label.
🔍 Why this matters: Semantic drift detectors are triage tools, not truth machines. Their job is to shrink the set of docs a human needs to read, not to replace the human. A 64% precision rate is useful if the alternative is reading 100% of the docs. It's not useful if you expect the tool to be right every time.
8. Treat OpenAPI Specs as a Separate Drift Surface
Scenario: The openapi.yaml file says POST /orders returns a 201 with an order_id string. The live API returns a 200 with an id integer. The spec was correct when it was written, but the implementation changed and nobody updated the spec. Every SDK generated from that spec is now wrong.
Why general doc drift tools miss this: OpenAPI drift isn't prose drift. It's a schema mismatch between a formal contract and a runtime behavior. You need to diff the contract against itself (across versions) and against live responses.
Solution: Use dedicated contract diffing. For version-to-version drift, tools like api-governor compare an OpenAPI spec against a baseline and classify breaking changes: removed endpoints, required parameters added, types narrowed . For spec-to-runtime drift, tools like barkapi hit live endpoints and diff the response shape against the spec, catching type changes, missing fields, nullability shifts, and undocumented additions .
# CI step: fail on breaking OpenAPI changes.
- name: OpenAPI breaking change check
run: |
pip install api-governor
api-governor openapi-v2.yaml --baseline openapi-v1.yaml --strict
api-governor exits non-zero on blockers like a removed endpoint or a newly required parameter. barkapi does the runtime half: it parses the spec, calls each endpoint, and reports mismatches with severity classification. Its exit code is 1 on breaking drift, which makes it a CI gate .
Why this works: OpenAPI drift has a formal structure that general-purpose doc tools can't exploit. A schema diff knows that string → integer is a breaking change and that adding an optional field is not. That precision is what makes contract diffing reliable enough to block a merge.
💡 Practical note: Run the spec-to-spec diff on every PR that touches the spec, and the spec-to-runtime diff on a schedule or in staging. Runtime diffing needs a live endpoint, so it's not a pre-merge gate unless you have a staging environment you can hit safely.
9. Wire It Into CI Without Drowning in False Positives
Scenario: You add a drift detector to CI. The first week, it fails on 30 pull requests, most of them for docs that were already stale before the PR. Developers start adding --no-verify or disabling the check. By the second week, the gate is gone.
Why CI gates for docs fail: A drift detector that reports pre-existing issues alongside new ones is indistinguishable from a broken gate. Developers can't tell what they caused from what was already there, so they route around the check.
Solution: Gate on deltas, not absolutes. The CI check should fail only when a PR introduces new drift or leaves a touched pair unacknowledged. Pre-existing findings stay in a baseline report and are triaged separately.
staledocs supports this with staged checks: staledocs check --gate strict in CI, plus a pre-commit hook that runs impact --staged . The staged variant only checks pairs affected by the current diff, which keeps the feedback loop fast and relevant.
# GitHub Actions: docs drift gate
name: Docs Drift
on: [pull_request]
jobs:
drift:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install staledocs
- name: Check only pairs touched by this PR
run: staledocs check --gate strict --staged
For LLM-based checks, run them on a schedule or as an advisory comment, not as a blocking gate. A semantic audit that takes 20 seconds per function and produces 36% false positives is not a merge blocker. It's a backlog generator.
Why this works: The gate enforces the invariant that matters: this PR must not make the repository less trustworthy than it already was. That's a claim developers can reason about and fix. “The repository has 200 drift findings” is not.
🚨 Production warning: Do not let the drift detector auto-fix documentation. An auto-fix that rewrites a doc to match the code can erase a deliberate spec that the code hasn't implemented yet. In staledocs terms, that's the CODE_LAG case—the doc is ahead, not behind. Auto-fixing it would destroy the specification. Detect, report, and let a human or an agent decide which side is wrong.
What I'd Build, What I'd Skip, and the Decision That Matters
If I were setting up drift detection for a repository in 2026, here's the stack and the sequence I'd use.
| Layer | Tool type | Why | When to run |
|---|---|---|---|
| Stable ID binding | docsync-style directives | Eliminates line-number fragility; content rendered from live source | At authoring time; checked in CI |
| Pair ledger | staledocs-style fingerprints | Deterministic detection of one-sided changes | On every commit |
| Anchor liveness | Backtick extraction + grep | Catches renamed/removed identifiers with zero false positives | On every commit |
| Coverage gate | File-to-doc pairing check | Prevents silent undocumented code | In CI, after backlog triage |
| Contract diff | OpenAPI schema comparison | Catches API surface drift precisely | On spec changes; runtime check in staging |
| Semantic audit | LLM claim extraction + verification | Catches meaning-level drift that identifiers can't | Nightly or pre-release |
| Execution check | Doc-derived test generation | Verifies behavioral claims by running them | For well-structured docstrings and API references |
What I'd skip
Running an LLM on every commit. The cost and latency aren't justified. Most commits don't change semantic meaning in a way that identifiers miss. Run the deterministic layer on every commit and the semantic layer on a schedule.
Treating “code has undocumented features” as drift. It's a coverage gap, not a lie. Conflating the two inflates the finding count and makes the tool feel noisy. Track coverage separately, and filter under-promises out of the drift report.
Auto-fixing docs to match code. A doc that describes an unimplemented feature is a specification, not a bug. Auto-fixing it destroys the intent. Report the mismatch and let a human decide which side is authoritative.
Building a custom detector from scratch before trying the existing tools. The 2026 ecosystem has mature options for every layer: docsync for stable IDs, staledocs for deterministic ledgers, LyingDocs for agent-based claim verification, CASCADE for execution-based checks, and api-governor/barkapi for OpenAPI drift. Start with the tools that match your failure modes, and only build custom logic for the gaps.
The decision that matters
The difference between a drift detector that survives and one that gets disabled is not the model or the parsing library. It's whether the tool's findings are actionable and attributable.
A finding that says “docs/auth.md line 34 references build:prod, which was removed in commit a1b2c3” is actionable. A finding that says “this doc may be inconsistent” is not. A gate that fails only on drift introduced by the current PR is attributable. A gate that fails on a backlog of 200 pre-existing issues is not.
Start with the deterministic layer. Get it to zero false positives. Wire it into CI as a delta check. Then, and only then, add semantic analysis on top—not as a gate, but as a triage queue. The deterministic layer tells you what broke. The semantic layer tells you what might be a lie. You need both, but in that order.