The first version of my repository RAG answered questions correctly and cited nothing useful.
It would say, “Authentication is handled in the middleware layer,” and link to src/. That's technically a citation. It's also worthless when you're staring at a 40,000-line codebase at 11 p.m. trying to verify whether a JWT is validated before or after the route handler runs.
The fix wasn't a better prompt. It was rebuilding the retrieval pipeline so that every chunk carries a file path, a symbol name, and an exact line range—and then forcing the generator to use those coordinates instead of paraphrasing them away. That distinction—between RAG that retrieves code and RAG that can prove where the code lives—is the entire engineering problem.
This is what I learned building it, and what I'd do differently if I started again in 2026.
📋 Table of Contents
- The Real Problem: Retrieval Without Coordinates
- 1. Chunk Code by Syntax, Not by Character Count
- 2. Hybrid Retrieval Is Not Optional for Code
- 3. The Reranker Is Where Precision Comes From
- 4. Follow the Graph, Not Just the Vector
- 5. Make the Model Cite Coordinates, Then Validate Them
- 6. Incremental Indexing: The Part Nobody Demos
- 7. Evaluate Retrieval and Citations Separately
- What I'd Build in 2026 (and What I'd Skip)
The Real Problem: Retrieval Without Coordinates
A generic document-RAG pipeline treats code like prose. It splits files every 500 tokens, embeds the slices, and retrieves the most semantically similar ones. For documentation, that works well enough. For code, it fails in a specific and predictable way.
Consider a question: “Where does the app attach the current user to the request object?”
A fixed-size chunker might return half of a function body, the closing brace of the previous function, and a comment from two functions ago. The embedding model sees tokens that look roughly relevant. The LLM sees a fragment with no signature, no surrounding control flow, and no indication of which file it came from. It generates a plausible answer. The citation is either missing or wrong.
The obvious fix is to include file names in the chunk metadata. That helps, but it doesn't solve the deeper issue: the model still doesn't know where the relevant code starts and ends. A file path alone tells you which file to open. It doesn't tell you which lines to read.
Exact line citations require three things to be true simultaneously:
- The chunk boundary must match a real syntactic unit. If a chunk starts in the middle of a function, the line numbers are still technically correct, but the context is wrong. The model cites a fragment.
- The retrieval system must carry line numbers as first-class metadata. Not as a string appended to the text, but as structured fields the prompt can reference and the application can resolve.
- The generator must be constrained to cite only what it retrieved. Left unconstrained, LLMs will invent line ranges that look plausible but don't correspond to the context.
The rest of this article is about making those three things hold in production.
🔍 Why this matters: A citation that points to the wrong lines is worse than no citation. It creates false confidence. Users stop verifying, and the tool becomes a hallucination amplifier with a professional-looking UI.
1. Chunk Code by Syntax, Not by Character Count
Scenario: You index a TypeScript repository with a standard recursive character splitter. A question about res.send() returns a chunk that begins halfway through the function body and ends after the closing brace of the next function. The model answers correctly, but cites lines 118–203 when the actual res.send declaration starts at line 111.
Why the obvious solution fails: Fixed-size splitting has no concept of declarations. It cannot distinguish a complete function from a fragment because it never parses the language.
Solution: Use a parser to split at syntactic boundaries. Tree-sitter is the pragmatic default in 2026 because it supports a wide range of languages, runs incrementally, and exposes byte ranges that map directly to line numbers. For TypeScript and JavaScript, the TypeScript Compiler API is also viable; for Python, the built-in ast module is enough.
Here's a minimal Tree-sitter chunker in Python that extracts functions and classes with exact line ranges:
from tree_sitter import Language, Parser
import tree_sitter_python as tspython
PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)
def chunk_python_file(source_code: str, file_path: str):
tree = parser.parse(source_code.encode("utf-8"))
root = tree.root_node
chunks = []
def visit(node):
if node.type in ("function_definition", "class_definition"):
start_line = node.start_point[0] + 1 # Tree-sitter is 0-indexed
end_line = node.end_point[0] + 1
# The node's text is the complete declaration, including decorators
# and the body, because Tree-sitter's range covers the whole subtree.
chunk_text = source_code[node.start_byte:node.end_byte]
name_node = node.child_by_field_name("name")
symbol_name = (
source_code[name_node.start_byte:name_node.end_byte]
if name_node else "<anonymous>"
)
chunks.append({
"text": chunk_text,
"metadata": {
"file_path": file_path,
"symbol": symbol_name,
"node_type": node.type,
"start_line": start_line,
"end_line": end_line,
},
})
for child in node.children:
visit(child)
visit(root)
return chunks
Why this works: The chunk text is exactly the source bytes of the function or class. The start_line and end_line come from the same node that produced the text, so they cannot drift independently. The symbol name gives the generator a human-readable anchor that survives embedding compression.
For a real repository, you also need to handle decorators, nested functions, and module-level assignments. A function nested inside another function will be visited twice if you're not careful—once as part of its parent's chunk and once as its own chunk. That's often acceptable, because both views are syntactically complete. It's when you start slicing function bodies into arbitrary windows that retrieval quality collapses.
💡 Practical note: Don't discard module-level code. In Express-style codebases, a huge amount of behavior lives in prototype assignments (
app.use = function...,proto.handle = function...) that aren't class methods or standard declarations. A chunker that only looks forclassandfunctionkeywords will miss over 80% of Express's core logic. Walk the AST forBinaryExpressionassignments too.
2. Hybrid Retrieval Is Not Optional for Code
Scenario: A user asks, “Where is parseFlow() defined?” The embedding model returns a chunk about a completely different function because parseFlow is a rare identifier that the model has never seen. Meanwhile, BM25 finds the exact declaration immediately.
Why dense-only retrieval fails here: Code identifiers are low-frequency tokens. Embedding models are trained to compress semantic meaning, and a rare camelCase symbol carries almost no distributional signal. The model treats it as noise. Exact-match lexical retrieval does the opposite: it excels at rare identifiers precisely because they're rare.
By 2026, hybrid retrieval is the de facto default for production RAG, and the evidence is not subtle. Public benchmarks consistently show BM25 plus dense retrieval plus a reranker outperforming dense-only retrieval by meaningful margins on entity-heavy and code-heavy corpora. Pure dense retrieval has largely been retired for serious code search because its failure modes on identifiers and codes are too expensive at scale.
The standard fusion method is Reciprocal Rank Fusion (RRF). It combines ranked lists by rank position rather than by score, which avoids the problem of normalizing cosine similarity against BM25 scores that live on completely different scales.
-- Postgres + pgvector: hybrid search with RRF in a single query.
-- Assumes a `chunks` table with `embedding vector(1536)`,
-- `content tsvector`, and `metadata jsonb`.
WITH dense AS (
SELECT id, metadata,
ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
FROM chunks
ORDER BY embedding <=> $1
LIMIT 50
),
lexical AS (
SELECT id, metadata,
ROW_NUMBER() OVER (
ORDER BY ts_rank_cd(content, plainto_tsquery($2)) DESC
) AS rank
FROM chunks
WHERE content @@ plainto_tsquery($2)
ORDER BY ts_rank_cd(content, plainto_tsquery($2)) DESC
LIMIT 50
)
SELECT
COALESCE(d.id, l.id) AS id,
COALESCE(d.metadata, l.metadata) AS metadata,
-- RRF with k = 60, the value from the original SIGIR 2009 paper.
COALESCE(1.0 / (60 + d.rank), 0.0)
+ COALESCE(1.0 / (60 + l.rank), 0.0) AS rrf_score
FROM dense d
FULL OUTER JOIN lexical l ON d.id = l.id
ORDER BY rrf_score DESC
LIMIT 10;
Doing fusion inside Postgres means one round trip instead of fetching two ranked lists and merging them in application code. RRF also sidesteps a subtle calibration problem: cosine distance and ts_rank_cd are not comparable quantities, so any weighted sum would require tuning a weight per corpus. Rank-based fusion avoids that entirely.
For code specifically, the lexical side needs identifier-aware tokenization. parseFlow should be indexed as parseflow, parse, and flow. Without that, BM25 still works, but it loses the ability to match partial identifiers and snake_case variants. A simple pre-tokenizer that splits on camelCase and underscores before inserting into the tsvector column is worth the effort.
⚠️ Gotcha: Postgres full-text search is not BM25. It uses
ts_rank_cd, which has no IDF weighting. For code retrieval where rare identifiers matter enormously, this is a real limitation. If you need true BM25, use a dedicated lexical engine like Elasticsearch or OpenSearch alongside your vector store—or accept the trade-off and rely on the reranker to fix the ranking.
3. The Reranker Is Where Precision Comes From
Scenario: Hybrid retrieval returns 20 candidates. The correct chunk is ranked 7th because the dense side pulled in semantically related but structurally irrelevant results. Without reranking, you pass the 7th result to the generator and hope.
Why this happens: Bi-encoder embeddings compress the query and the document independently. They never see each other. The similarity score is an approximation. A cross-encoder reads the query and the candidate together and produces a relevance score conditioned on both. It's slower—you can't precompute cross-encoder scores for the whole corpus—but it's dramatically more precise on the top candidates.
The production pattern is two-stage retrieval: retrieve a candidate pool of 20 to 100 with hybrid search, then rerank that pool with a cross-encoder and keep the top 5 to 10.
In 2026, there are code-specific rerankers that outperform general-purpose models. The code-daemon-reranker-v1 is a 117M-parameter XLM-RoBERTa variant fine-tuned on natural-language-to-code pairs, with a reported +0.040 nDCG@10 over the retriever alone and +0.010 over a public ranker of the same width. Those are single-system numbers on a specific candidate distribution, not a universal benchmark, but the direction is consistent: a small, domain-tuned cross-encoder is a better investment than a larger general reranker.
Here's how to call one in an ONNX runtime, which is the practical choice for local or self-hosted deployment:
import onnxruntime as ort
import numpy as np
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("faxenoff/code-daemon-reranker-v1")
session = ort.InferenceSession(
"model.onnx",
providers=["CPUExecutionProvider"],
)
def rerank(query: str, candidates: list[dict], top_k: int = 8):
pairs = [(query, c["text"]) for c in candidates]
encoded = tokenizer.batch_encode_plus(
pairs,
padding=True,
truncation="longest_first",
max_length=256,
return_tensors="np",
)
logits = session.run(
None,
{
"input_ids": encoded["input_ids"].astype(np.int64),
"attention_mask": encoded["attention_mask"].astype(np.int64),
},
)[0].squeeze(-1)
ranked = sorted(
zip(candidates, logits),
key=lambda pair: float(pair[1]),
reverse=True,
)
return [c for c, _ in ranked[:top_k]]
The 256-token limit is worth noticing. The query is kept whole, but long code candidates get truncated at the tail. That's fine for function-level chunks, which are usually under 200 tokens. It becomes a problem if you chunk by entire files or large classes. Another argument for syntax-aware chunking.
🧠 The important part: Reranking doesn't fix bad recall. If the correct chunk isn't in the candidate pool, no reranker can recover it. Improve recall first (hybrid retrieval, identifier tokenization, graph expansion), then use the reranker for precision.
4. Follow the Graph, Not Just the Vector
Scenario: A user asks, “How does the login flow reach the database?” The retriever finds the route handler and the database query function separately, because both are semantically related to the question. But it misses the three intermediate functions that actually connect them. The generator sees two disconnected chunks and either guesses at the connection or produces an incomplete answer.
Why vector search struggles here: The connection between the route handler and the query function is structural, not semantic. The embedding of the route handler doesn't necessarily resemble the embedding of the query function, even though one calls the other through a chain of imports and function calls. Vector similarity captures meaning, not reachability.
The solution is to build a lightweight code graph alongside the vector index. At minimum, the graph needs nodes for files, symbols, and the call edges between them. Tree-sitter can extract function calls and imports during the same parse pass that produces chunks. You don't need a full program analysis—a one-hop or two-hop expansion from the initial retrieval results is enough to catch most of the missing context.
A pragmatic implementation in 2026 uses an adjacency table in the same Postgres database:
CREATE TABLE code_edges (
source_symbol TEXT NOT NULL,
source_file TEXT NOT NULL,
target_symbol TEXT NOT NULL,
target_file TEXT,
edge_type TEXT NOT NULL, -- 'calls', 'imports', 'contains'
PRIMARY KEY (source_file, source_symbol, target_symbol, edge_type)
);
-- After retrieving top-k chunks, expand one hop along call edges.
WITH seed AS (
SELECT metadata->>'symbol' AS symbol,
metadata->>'file_path' AS file_path
FROM hybrid_search_results
LIMIT 5
)
SELECT c.*
FROM chunks c
JOIN code_edges e
ON e.target_symbol = c.metadata->>'symbol'
AND e.target_file = c.metadata->>'file_path'
WHERE e.source_symbol IN (SELECT symbol FROM seed)
AND e.edge_type = 'calls'
LIMIT 10;
The expansion results go into the same context window as the original retrieval results, but with a lower priority. If the token budget is tight, drop the graph-expanded chunks first. Their job is to fill structural gaps, not to dominate the context.
A full knowledge graph in Memgraph or Neo4j is also viable, and projects like code-graph-rag use that approach for multi-hop queries and impact analysis. But for a GitHub repository RAG that answers where and how questions, a one-hop adjacency table is usually enough. The jump from zero graph context to one-hop context is large. The jump from one-hop to three-hop is much smaller, and the complexity cost is not always justified.
💡 Practical note: Graph expansion is most valuable for “how does X reach Y” questions and least valuable for “where is X defined” questions. Route the query through a lightweight planner, or simply always expand but weight graph-expanded chunks lower in the final context ordering.
5. Make the Model Cite Coordinates, Then Validate Them
Scenario: The retrieval pipeline returns five chunks, each with file_path, symbol, start_line, and end_line. The prompt says “cite your sources.” The model responds with a correct answer and a citation to src/auth/middleware.ts:38–55. The citation is correct. But on the next question, it cites src/auth/middleware.ts:42–60, which is a plausible-looking range that doesn't match any retrieved chunk.
Why this happens: LLMs are good at producing citation-shaped text and unreliable at reproducing exact coordinates from context. The model knows it should cite something, so it generates a range that looks reasonable. Without validation, the UI renders a link that points to the wrong lines.
The fix is to separate what the model is allowed to cite from how the citation is rendered. The model should emit a stable source ID—[1], [2], [3]—that maps to a retrieved chunk. The application resolves that ID to the file path and line range. This is the same pattern OpenAI's citation guidance recommends: let the model emit source IDs, and let your system resolve or render the precise locator.
Here's the prompt structure that worked reliably:
You are answering questions about a GitHub repository.
Use ONLY the numbered sources below. Do not use outside knowledge.
Each source has a stable ID and exact coordinates. When you make a claim,
end the sentence with the ID in square brackets, like [2].
If the sources do not contain enough information, say "The retrieved code
does not answer this question." Do not guess.
Sources:
[1] File: lib/response.js
Symbol: res.send
Lines: 111-236
Type: prototype_method
---
<source code>
[2] File: lib/application.js
Symbol: app.use
Lines: 45-92
Type: prototype_method
---
<source code>
After generation, a post-processor extracts every [N] marker and checks it against the retrieved source IDs. If the model cites [7] but only five sources were provided, the response is rejected or regenerated. If it makes a factual claim with no citation, the claim is flagged.
The UI then renders each [N] as a link to the exact lines:
function renderCitation(sourceId: number, sources: Source[]) {
const source = sources.find((s) => s.id === sourceId);
if (!source) return "[invalid citation]";
const { file_path, start_line, end_line, symbol } = source.metadata;
const url = `https://github.com/${repo}/blob/${commit}/${file_path}#L${start_line}-L${end_line}`;
return `[${sourceId}](${url}) (${file_path}:${start_line}–${end_line}, ${symbol})`;
}
The commit in the URL matters. If you link to main, the line numbers drift as soon as someone pushes a commit. Pinning citations to the indexed commit makes them stable and reproducible.
🚨 Production warning: Do not let the model generate line ranges directly, even if it seems to do a good job in testing. The failure mode is silent and intermittent. A model that produces correct line ranges 95% of the time is worse than a model that produces source IDs 100% of the time, because the 5% failures are indistinguishable from successes until a user clicks the link.
6. Incremental Indexing: The Part Nobody Demos
Scenario: The first index takes 12 minutes on a medium-sized repository. A developer pushes a commit that changes three files. The demo re-indexes the entire repository. In production, this happens dozens of times a day, and the index is never quite in sync with main.
Why full re-indexing fails in production: Embedding is the expensive step. Re-embedding unchanged files is pure waste. At scale, every serious system uses delta-based updates—processing only changed files rather than re-indexing the entire corpus.
The simplest reliable approach is content-addressed hashing. Store the Git blob SHA for each file alongside the chunks. On each indexing run, compare the current blob SHA to the stored value. If they match, skip the file. If they differ, delete the old chunks for that file and insert new ones.
import subprocess
from pathlib import Path
def changed_files(repo_path: str, last_indexed_commit: str) -> list[str]:
"""Return files changed between the last indexed commit and HEAD."""
result = subprocess.run(
["git", "-C", repo_path, "diff", "--name-only",
f"{last_indexed_commit}..HEAD"],
capture_output=True,
text=True,
check=True,
)
return [line for line in result.stdout.splitlines() if line.strip()]
def incremental_index(repo_path: str, last_commit: str):
files = changed_files(repo_path, last_commit)
for relative_path in files:
absolute_path = Path(repo_path) / relative_path
if not absolute_path.suffix in {".py", ".ts", ".js", ".go", ".rs"}:
continue
if not absolute_path.exists():
delete_chunks_for_file(relative_path)
continue
source = absolute_path.read_text(encoding="utf-8")
chunks = chunk_python_file(source, relative_path) # or language-specific
delete_chunks_for_file(relative_path)
embed_and_insert(chunks)
For larger repositories, Git's own object model gives you a head start. The blob SHA is a content hash. If the SHA hasn't changed, the file content is identical. You don't even need to read the file. Tools like Cursor and Meta Glean use Merkle trees and change-tracking databases to make this efficient at monorepo scale, but a straightforward git diff plus blob SHA comparison covers most GitHub repository use cases.
One subtlety: renames. A renamed file has the same blob SHA but a different path. If your chunk metadata stores the file path, a rename requires updating the metadata even though the content is unchanged. Handle renames explicitly with git diff --find-renames, or accept a small amount of stale metadata until the next full reconciliation.
💡 Practical note: The index doesn't have to be real-time. For a code Q&A tool, a delay of one commit is usually acceptable. What matters is that the index is consistent: every chunk in the index corresponds to a known commit, and citations resolve against that commit. Eventual consistency with a clear commit boundary is better than a half-updated index that mixes files from different revisions.
7. Evaluate Retrieval and Citations Separately
Scenario: The demo looks great. Users ask questions, the model answers, citations appear. Then someone asks a question whose answer spans two functions, and the model cites only one. Or a question about a deprecated function returns the new implementation with confident citations to the wrong file.
Why “it looks good” is not an evaluation: RAG failures are distributed across two stages. Retrieval can fail (the correct chunk isn't in the context), or generation can fail (the chunk is there, but the model ignores it or cites it incorrectly). If you only evaluate the final answer, you can't tell which stage is broken.
The minimum viable evaluation has two layers:
Retrieval metrics. For a set of golden questions, annotate the expected file and line range. Then measure:
- Hit@k: Is the correct chunk in the top-k retrieved results?
- MRR@k: How high is the correct chunk ranked?
- File-level precision: What fraction of retrieved chunks are from the expected file?
A useful sanity check from the code retrieval literature: AST-based symbol retrieval has been shown to cut the amount of code context sent to the LLM by 1.6× to 3.9× compared with naive chunking, while maintaining or improving answer quality. That's a measurable claim you can reproduce on your own corpus: compare tokens-sent-per-correct-answer between fixed-size chunking and syntax-aware chunking.
Citation metrics. For each generated answer, measure:
- Citation validity: Does every
[N]map to a retrieved source? - Citation correctness: Does the cited source actually contain the claim?
- Coverage: Are there factual claims without citations?
Citation correctness is the hardest to automate. A practical approach is an LLM-as-judge with a strict rubric: given the claim and the cited chunk, does the chunk support the claim? This is not perfect, but it catches the obvious failures—citing a comment when the claim is about a function body, or citing a function signature when the claim is about a return value.
The evaluation harness should fail CI when retrieval quality regresses. A small golden.jsonl file with 30 to 50 questions, expected files, and expected line ranges is enough to catch most regressions:
{"question": "Where is JWT verification performed?", "expected_file": "backend/middleware.py", "expected_lines": [38, 55]}
{"question": "How does res.send set the Content-Length header?", "expected_file": "lib/response.js", "expected_lines": [111, 236]}
{"question": "Which function attaches the user to the request object?", "expected_file": "lib/request.js", "expected_lines": [40, 72]}
Run this on every indexing pipeline change. If Hit@5 drops by more than a few percentage points, block the merge. Retrieval regressions are silent in manual testing and obvious in CI.
🔍 Why this matters: The quality of a RAG system depends more on retrieval than on the LLM. A frontier model with bad chunks will produce confident, well-written, wrong answers. A mid-tier model with precise, syntactically complete chunks and correct coordinates will produce answers that are easy to verify. Invest in retrieval evaluation before you invest in prompt engineering.
What I'd Build in 2026 (and What I'd Skip)
If I were starting this project from scratch today, here's the architecture I'd choose, and the parts I'd deliberately leave out.
The stack I'd actually use
| Layer | Choice | Why |
|---|---|---|
| Parsing | Tree-sitter | Broad language support, incremental parsing, byte ranges map to line numbers. Avoids a language-specific parser per ecosystem. |
| Chunking | AST nodes only | Function, method, class, and module-level assignment boundaries. No fixed-size fallback except for non-code files like Markdown and YAML. |
| Lexical index | Postgres full-text or a dedicated BM25 engine | Start with Postgres to keep the stack small. Move to OpenSearch only if identifier recall is measurably insufficient. |
| Vector index | pgvector with HNSW | Persistent, transactional, and good enough for repository-scale corpora. Avoid FAISS unless you're comfortable rebuilding the index on every restart. |
| Embeddings | A code-aware model | General text embeddings are acceptable. Code-specific models like LateOn-Code or Qwen3-Embedding variants are worth benchmarking on your own retrieval set, but don't assume vendor leaderboard rankings transfer to your queries. |
| Reranker | Small cross-encoder, code-tuned | A 100–600M parameter cross-encoder reranking the top 20–50 candidates. This is the highest-leverage model in the pipeline for precision. |
| Graph | One-hop adjacency table | Calls and imports, extracted during parsing. Skip the full graph database unless you need multi-hop impact analysis. |
| Generation | Any frontier model with source-ID citation constraints | The model matters less than the prompt structure and the post-validation. |
| Evaluation | Golden questions + CI gate | Retrieval metrics and citation validity, run on every pipeline change. |
What I'd skip
Full-file embedding. Embedding whole files dilutes the signal. A 2,000-line file has one vector, and that vector represents everything and nothing. Function-level chunks are the right granularity for code retrieval.
A generic document chunker. Recursive character splitting is fine for prose. It is actively harmful for code. The first time a chunk starts in the middle of a conditional and ends after the closing brace of an unrelated function, you'll understand why.
LLM-generated summaries as the only index. Summaries are useful as a supplementary retrieval signal—they can help match natural-language questions to code that doesn't use the question's vocabulary. But they should not replace the raw source chunk. The source chunk is what the model needs to generate an accurate answer and cite exact lines.
Trusting the model to produce line ranges. Use source IDs. Resolve coordinates in your application. Validate every citation before rendering.
The decision that matters most
The difference between a repository RAG that feels like a toy and one that feels like a tool is not the LLM. It's whether the retrieved context is structurally complete and spatially anchored.
A complete function with a file path and a line range gives the model everything it needs to answer and cite. A 500-token slice with a file name gives the model fragments and hope.
If you get the chunking and the coordinates right, the rest of the pipeline becomes much more forgiving. If you get them wrong, no amount of reranking or prompt engineering will save the citations. Build the index so that every piece of code knows exactly where it lives, and the answers will follow.