The index you built six months ago is now a liability.
Not because the embedding model was wrong. Not because the chunk size was off. Because the documents changed, and the index didn't. A pricing page was updated. An API endpoint was deprecated. A policy was superseded. The vector store still serves the old chunks with the same confidence it served them on day one, and the model has no way to know that the answer it's grounding on was retired three quarters ago.
This is the defining failure of RAG in production, and it isn't a retrieval problem. It's a freshness problem. Standard RAG evaluation benchmarks don't catch it, because faithfulness measures whether the answer stayed within the retrieved context — not whether that context was current. Most teams discover the gap from user complaints, not from a metric.
The platform that survives this is not a better pipeline. It's a system where ingestion is continuous, retrieval is agentic, and the index is treated as a living artifact that can be audited, scored, and repaired. Python handles the parsing, chunking, and agent logic. PostgreSQL with pgvector handles storage and hybrid search in a single engine. n8n handles the event-driven glue that keeps the corpus aligned with reality. The AI agents handle the reasoning about what to retrieve, what to trust, and when to say "I don't know."
📋 Table of Contents
- 1. The Staleness Gap: Why Your RAG Demo Fails Six Months Later
- 2. Ingestion Is Where Production RAG Actually Breaks
- 3. Chunking by Semantic Unit, Not by Token Count
- 4. The PostgreSQL Schema That Survives an Update
- 5. Hybrid Retrieval and Reranking: Cosine Alone Is Not Enough
- 6. Agentic RAG: Corrective, Self-Reflective, and Adaptive Retrieval
- 7. The Self-Updating Loop: n8n, Webhooks, and Incremental Re-Indexing
- 8. Freshness Scoring: Measuring the Thing That Actually Breaks
- 9. Evaluation: Faithfulness, Context Recall, and the Metrics That Matter
- 10. Permissions and Provenance: Who Is Allowed to Know What
- The Build Order: What to Ship First and What to Skip
1. The Staleness Gap: Why Your RAG Demo Fails Six Months Later
Scenario: You build a RAG assistant over your product documentation. The demo works. You answer questions about the API, the pricing, and the onboarding flow. Six months later, the API has two new endpoints, the pricing page has changed twice, and a deprecated endpoint is still in the index with a high cosine similarity to queries about the replacement. The assistant confidently recommends the deprecated endpoint. The user follows the recommendation. It fails.
The obvious fix is to re-index nightly. But a nightly batch re-index creates up to a 24-hour staleness window, and during that window the model answers questions based on outdated context. For a knowledge base that changes daily — documentation, support articles, pricing, policies — nightly re-indexing is a design failure in 2026. The cost is also wrong: re-embedding the entire corpus every night treats an unchanged page and a rewritten page as the same problem, and you pay to chunk and embed both.
The deeper problem is that RAG systems treat all content as semantically equivalent. A chunk from a 2019 document and a chunk from yesterday's update sit in the same vector space, and the retrieval model has no concept of which one is authoritative. Real knowledge evolves — guidance gets retracted, APIs change version, policies supersede each other — and a static index produces stale and contradictory answers.
The platform has to solve three distinct problems:
- Freshness: the index must reflect the current state of the source, with a measurable lag that is small relative to the rate of change.
- Correctness: retrieval must prefer current, authoritative content over stale or superseded content, even when the stale content is semantically similar.
- Confidence: the system must know when it doesn't know, and say so instead of grounding an answer in a document that was edited three weeks ago.
The architecture that addresses all three is not a single component. It's a loop: ingest, chunk, embed, index, retrieve, evaluate, and repair. The loop runs continuously, and every stage produces a signal that the next stage can use.
🔍 Why this matters: The freshness of a RAG system is not a property of the embedding model. It's a property of the pipeline that maintains the index. A perfect embedding of a stale document is still a stale answer.
2. Ingestion Is Where Production RAG Actually Breaks
Scenario: Your parser reads a PDF and returns the body text, but it also returns the headers, the footers, the page numbers, and the table of contents. The chunker splits the text into 512-token windows, and one window contains the footer of page 4, the header of page 5, and the first paragraph of a section. The embedding for that chunk represents a mixture of navigation chrome and content. The retriever returns it for a query about the section, and the model gets a chunk that starts mid-sentence and ends with a copyright notice.
The research consensus is blunt: most RAG failures trace back to the ingestion and chunking layer, not the LLM. If the parser returns garbage, the chunks are garbage, the embeddings are garbage, and the retrieval returns garbage, regardless of how well everything downstream is configured.
The fix starts with layout-aware parsing. For PDFs, use a parser that distinguishes body text from headers, tables, and figures — Unstructured.io or LlamaParse rather than PyPDF2 or pdfminer directly. For HTML, strip navigation, sidebars, and footers before converting to Markdown. For scanned documents, run OCR and validate the output against a sample of known text.
The second fix is normalization. The same document may arrive in multiple formats — PDF, HTML, Markdown — and the pipeline should produce a canonical representation. Normalize whitespace, strip control characters, and collapse repeated line breaks. The content hash should be computed over the normalized text, not the raw bytes, so a formatting change doesn't trigger a re-embed.
The third fix is metadata. Every document should carry a source_url, a last_modified timestamp, a content_hash, and a version_label where applicable. The metadata is not decoration; it's the input to the freshness scoring and the retrieval filter. A chunk without a source timestamp cannot be audited, and an unauditable chunk is a liability.
A minimal Python ingestion pipeline that does all three:
from datetime import datetime, timezone
from hashlib import sha256
from pathlib import Path
import re
from unstructured.partition.auto import partition
def normalize(text: str) -> str:
text = re.sub(r"\r\n?", "\n", text)
text = re.sub(r"[ \t]+", " ", text)
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip()
def ingest(path: Path) -> dict:
elements = partition(filename=str(path), strategy="fast")
body = "\n\n".join(
str(el) for el in elements
if el.category not in {"Header", "Footer", "PageNumber"}
)
normalized = normalize(body)
content_hash = sha256(normalized.encode("utf-8")).hexdigest()
return {
"source_path": str(path),
"source_url": path.as_uri(),
"content": normalized,
"content_hash": content_hash,
"last_modified": datetime.fromtimestamp(
path.stat().st_mtime, tz=timezone.utc
).isoformat(),
"ingested_at": datetime.now(timezone.utc).isoformat(),
}
The partition() call returns typed elements, and the pipeline filters out the categories that are navigation chrome. The content_hash is computed over the normalized text, which means a whitespace-only change doesn't invalidate the embedding. The last_modified timestamp is the source's modification time, not the ingestion time; the two are different signals, and conflating them makes freshness scoring meaningless.
💡 Practical note: Store the raw parsed text alongside the normalized text. When a retrieval result looks wrong, the first question is whether the parser or the retrieval is at fault. Without the raw text, you can't answer that question without re-running the parser.
3. Chunking by Semantic Unit, Not by Token Count
Scenario: You chunk a contract by 512 tokens. The chunk that contains the termination clause starts in the middle of the payment terms and ends in the middle of the liability section. The retriever returns it for a question about termination, and the model sees the clause without its heading, its sub-clauses, or the definition of "Term" that the clause depends on. It produces an answer that is syntactically correct and legally wrong.
Fixed-size chunking breaks RAG by splitting semantic units — tables, clauses, function definitions — across boundaries. The chunking strategy should match the document type: hierarchical for multi-page contracts, semantic for heterogeneous documents, fixed-size only for uniform corpora where the structure is genuinely flat.
For structured documents, chunk at the section boundary. A contract has clauses; a policy has sections; an API reference has endpoints. Each chunk should include the heading hierarchy that gives it context — the section title, the parent section, and the document title — so the model can see where the chunk sits.
For unstructured documents, use semantic chunking: split at paragraph boundaries, then merge adjacent paragraphs until the chunk reaches a target size. The target size is a function of the embedding model's context window and the retrieval granularity you want. A 256-token chunk retrieves more precisely but loses context; a 1,024-token chunk preserves context but retrieves more noise. The right answer depends on the query type: factual lookups want smaller chunks, explanatory questions want larger ones.
The non-obvious detail is that the chunk should store its parent context separately from its content. The embedding is computed over the chunk content plus a short context prefix, but the retrieved payload includes the full parent section. This is the "small-to-big" pattern: retrieve on the precise chunk, generate on the surrounding context.
from dataclasses import dataclass
@dataclass
class Chunk:
doc_id: str
chunk_index: int
content: str
context_prefix: str
parent_section: str
token_count: int
def chunk_hierarchical(doc: dict, max_tokens: int = 512) -> list[Chunk]:
chunks = []
for section in doc["sections"]:
paragraphs = section["paragraphs"]
current = []
current_tokens = 0
for para in paragraphs:
para_tokens = estimate_tokens(para)
if current_tokens + para_tokens > max_tokens and current:
chunks.append(Chunk(
doc_id=doc["id"],
chunk_index=len(chunks),
content="\n\n".join(current),
context_prefix=f"{doc['title']} > {section['title']}",
parent_section=section["id"],
token_count=current_tokens,
))
current, current_tokens = [], 0
current.append(para)
current_tokens += para_tokens
if current:
chunks.append(Chunk(
doc_id=doc["id"],
chunk_index=len(chunks),
content="\n\n".join(current),
context_prefix=f"{doc['title']} > {section['title']}",
parent_section=section["id"],
token_count=current_tokens,
))
return chunks
The context_prefix is prepended to the chunk before embedding, but the stored content remains the raw chunk. That way the embedding encodes both the local text and its position in the document, while the retrieval payload stays clean. The parent_section link lets the retrieval layer expand the context without a second query.
⚠️ Gotcha: Do not chunk tables with a text splitter. A table split across chunks loses its headers, and the model reads the values as unrelated numbers. Parse tables into structured rows, embed a serialized representation, and store the full table as a separate resource that the agent can fetch.
4. The PostgreSQL Schema That Survives an Update
The schema is where most RAG tutorials stop being useful. They show a single table with id, content, and embedding. That schema works for a demo and breaks under incremental updates, multi-source provenance, and hybrid retrieval.
The schema I'd run in production has four tables: documents, chunks, ingestion_runs, and retrieval_feedback. The documents table stores the canonical document with its source metadata and content hash. The chunks table stores the chunk content, its embedding, its tsvector for full-text search, and a foreign key to the document and the ingestion run that produced it. The ingestion_runs table records every pipeline execution, including what changed and what was skipped. The retrieval_feedback table records which chunks were retrieved for which queries and whether the answer was rated helpful, which is the signal the freshness loop uses to down-weight content that keeps producing bad answers.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE TABLE documents (
id BIGSERIAL PRIMARY KEY,
source_path TEXT NOT NULL,
source_url TEXT,
title TEXT,
content_hash CHAR(64) NOT NULL,
last_modified TIMESTAMPTZ,
last_verified TIMESTAMPTZ NOT NULL DEFAULT now(),
version_label TEXT,
is_deleted BOOLEAN NOT NULL DEFAULT false,
UNIQUE (source_path)
);
CREATE TABLE chunks (
id BIGSERIAL PRIMARY KEY,
document_id BIGINT NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
chunk_index INT NOT NULL,
content TEXT NOT NULL,
context_prefix TEXT,
embedding halfvec(1536),
embedding_model TEXT NOT NULL,
tsv TSVECTOR GENERATED ALWAYS AS (
to_tsvector('simple', coalesce(context_prefix, '') || ' ' || content)
) STORED,
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (document_id, chunk_index)
);
CREATE INDEX chunks_embedding_idx ON chunks
USING hnsw (embedding halfvec_cosine_ops)
WITH (m = 16, ef_construction = 64);
CREATE INDEX chunks_tsv_idx ON chunks USING GIN (tsv);
CREATE INDEX chunks_doc_idx ON chunks (document_id);
CREATE TABLE ingestion_runs (
id BIGSERIAL PRIMARY KEY,
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
finished_at TIMESTAMPTZ,
documents_seen INT,
documents_changed INT,
chunks_created INT,
chunks_deleted INT,
status TEXT NOT NULL DEFAULT 'running'
);
Several details are load-bearing.
halfvec(1536) instead of vector(1536) halves storage and index size, and pgvector 0.8 supports HNSW indexing on halfvec up to 4,000 dimensions. The 2026 default for RAG workloads is HNSW + halfvec, not full-precision vectors.
tsv uses the simple text search configuration, not english. Code, product names, and exact identifiers should not be stemmed. The pg_trgm extension is available for substring matching when needed.
The documents table separates content_hash from last_verified. The hash tells you whether the content changed; last_verified tells you when the source was last checked, even if it didn't change. A document can be verified daily without being re-embedded, and that distinction is what makes freshness scoring possible.
The ingestion_runs table is the audit trail. When a retrieval result looks wrong, the first query is "which ingestion run produced this chunk, and what did that run see?" Without that table, the only answer is "we don't know."
🧠 The important part: One Postgres for RAG, transactions, and analytics is a real architecture in 2026 — until you cross roughly 100 million vectors. Below that threshold, a separate vector database introduces a second source of truth and a synchronization problem that is worse than the marginal recall gain.
5. Hybrid Retrieval and Reranking: Cosine Alone Is Not Enough
Scenario: A developer asks "where is ERR_MODULE_NOT_FOUND thrown?" The vector index returns five chunks about error handling, logging, and module resolution. None of them contain the literal error code, because an embedding encodes meaning and a literal token like ERR_MODULE_NOT_FOUND holds no privileged position in the vector space. The answer is in a chunk that contains the exact string, and the dense retriever missed it.
Pure vector search fails on exact identifiers. Product SKUs, invoice numbers, function names, and error codes are not semantic; they're lexical. The fix is hybrid retrieval: run a dense retriever and a sparse retriever in parallel, then fuse the rankings.
Reciprocal Rank Fusion (RRF) is the standard fusion method. Each result gets a score of 1 / (k + rank) from each retriever, with k = 60 as the common default, so cosine distance and ts_rank_cd never have to be normalized against each other. The scores are summed, and the combined list is sorted by total score. A chunk that appears in both lists is promoted above a chunk that is strong in only one.
A 2026 benchmark on curriculum-aligned STEM textbooks found that hybrid retrieval (RRF) outperformed both lexical (BM25) and dense (FAISS) retrieval on recall and precision, and showed no recall drop on paraphrased queries while FAISS dropped the most. The numbers are domain-specific, but the direction is consistent: fusion beats either retriever alone.
WITH semantic AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
FROM chunks
WHERE embedding_model = $2
ORDER BY embedding <=> $1
LIMIT 40
),
lexical AS (
SELECT id, ROW_NUMBER() OVER (
ORDER BY ts_rank_cd(tsv, plainto_tsquery('simple', $3)) DESC
) AS rank
FROM chunks
WHERE tsv @@ plainto_tsquery('simple', $3)
LIMIT 40
),
fused AS (
SELECT
COALESCE(s.id, l.id) AS id,
COALESCE(1.0 / (60 + s.rank), 0) +
COALESCE(1.0 / (60 + l.rank), 0) AS score
FROM semantic s
FULL OUTER JOIN lexical l ON s.id = l.id
)
SELECT c.*, f.score
FROM fused f
JOIN chunks c ON c.id = f.id
ORDER BY f.score DESC
LIMIT 8;
The FULL OUTER JOIN is the part that matters. A chunk that appears only in the semantic results still gets a score. A chunk that appears only in the lexical results still gets a score. A chunk that appears in both gets the sum, which correctly promotes it.
The second stage is reranking. RRF gives you a good candidate set, but a cross-encoder reranker scores each candidate against the query directly and produces a more precise ordering. The reranker is slower and more expensive, so you run it on the top 20 candidates, not the whole index. The pattern is retrieve-then-rerank: hybrid retrieval for recall, cross-encoder for precision.
🚨 Production warning: Do not skip the reranker because the top-5 "looks right." A dense retriever optimized for recall returns candidates that are semantically adjacent but not precise. The reranker is what turns adjacent into relevant, and it is the single highest-leverage component after chunking.
6. Agentic RAG: Corrective, Self-Reflective, and Adaptive Retrieval
Classic RAG is a fixed pipeline: one retrieval, one generation. It fails when the query is ambiguous, when the retrieved context is insufficient, or when the answer requires combining information from multiple sources. Agentic RAG treats retrieval as a decision problem: the system decides whether to retrieve, what to retrieve, and whether the retrieved context is good enough before generating.
Three patterns have emerged, and they are complementary rather than competing.
Corrective RAG (CRAG) adds a lightweight retrieval evaluator that grades the retrieved documents and triggers a fallback — typically a web search — when the documents are poor. The evaluator is a small model or a classifier, not the generator, which keeps the cost low. In a 2026 comparative study, Corrective-RAG achieved the highest context precision (0.813) among the tested architectures.
Self-RAG trains the model to decide when to retrieve and to critique its own output for relevance and support. In the same study, Self-RAG was best on answer relevancy (0.784) and context recall (0.932). The trade-off is latency: self-reflection adds tokens and turns, and it is not the right choice for tight-latency applications.
Adaptive RAG routes each query to the retrieval strategy that fits its complexity. A simple factual question gets a single retrieval; a multi-hop question gets iterative retrieval; a question that the model can answer from its weights gets no retrieval at all. In the comparative study, adaptive-RAG was "adequate in terms of trade-off efficiency" — it didn't win on any single metric, but it spent the least compute per query.
The practical architecture combines them. A classifier routes the query by complexity. A retrieval critic grades the retrieved context and decides whether to retry, rewrite the query, or fall back. A generator produces the answer with citations. A faithfulness check verifies that the answer is supported by the retrieved context, and if it isn't, the system returns "I don't know" instead of hallucinating.
A minimal agentic retrieval loop in Python:
from dataclasses import dataclass
@dataclass
class RetrievalResult:
chunks: list[dict]
confidence: float
strategy: str
def agentic_retrieve(query: str, max_attempts: int = 3) -> RetrievalResult:
strategy = classify_query(query) # "direct" | "hybrid" | "multihop"
for attempt in range(max_attempts):
chunks = retrieve(query, strategy=strategy)
grade = critique_context(query, chunks)
if grade.score >= 0.7:
return RetrievalResult(chunks, grade.score, strategy)
if grade.reason == "insufficient":
query = rewrite_query(query, chunks)
strategy = "multihop"
elif grade.reason == "off_topic":
strategy = "hybrid"
return RetrievalResult(chunks, grade.score, f"{strategy}:exhausted")
The critique_context() function is the key. It asks a small model: "Do these chunks contain enough information to answer the query?" If the answer is no, the loop rewrites the query and retries. If the answer is yes, the loop returns the chunks with a confidence score. The confidence score is what the generation layer uses to decide whether to answer or to escalate.
🔍 Why this matters: Agentic RAG is not about giving the model more tools. It's about giving the system a way to fail gracefully. A fixed pipeline that retrieves the wrong context and generates anyway is worse than an agentic pipeline that retrieves, checks, and says "I don't have enough information."
7. The Self-Updating Loop: n8n, Webhooks, and Incremental Re-Indexing
The freshness problem is an event problem, not a schedule problem. A document changes when someone edits it, not when the cron job runs. The pipeline should react to the change event, not to the clock.
n8n is the right tool for the event glue because it already speaks to the systems where changes happen: GitHub, Notion, Google Drive, Confluence, Slack, and generic webhooks. The pattern is a webhook trigger that receives a change event, a normalization node that extracts the source and the content hash, a conditional node that checks whether the hash matches the indexed version, and an HTTP Request node that calls the Python ingestion service only when the content actually changed.
The n8n workflow doesn't do the parsing or the embedding. It does the orchestration: receive, normalize, compare, dispatch, and record. The Python service does the heavy lifting. The separation matters because n8n is good at connecting systems and bad at CPU-bound text processing, and the Python service is the opposite.
A minimal n8n workflow definition (simplified JSON):
{
"nodes": [
{
"name": "Webhook",
"type": "n8n-nodes-base.webhook",
"parameters": { "path": "doc-changed", "method": "POST" }
},
{
"name": "Normalize",
"type": "n8n-nodes-base.code",
"parameters": {
"jsCode": "return items.map(i => ({ json: { source: i.json.source, url: i.json.url, hash: i.json.content_hash } }));"
}
},
{
"name": "Changed?",
"type": "n8n-nodes-base.if",
"parameters": {
"conditions": {
"string": [{ "value1": "={{ $json.hash }}", "operation": "notEqual", "value2": "={{ $json.indexed_hash }}" }]
}
}
},
{
"name": "Reindex",
"type": "n8n-nodes-base.httpRequest",
"parameters": {
"method": "POST",
"url": "http://ingestion:8000/reindex",
"body": { "source": "={{ $json.source }}", "url": "={{ $json.url }}" }
}
}
]
}
The Changed? node is the cost control. If the hash matches, the workflow exits without calling the ingestion service. On a real corpus, most change events are no-ops — a file was touched but not modified, a page was re-saved without edits — and skipping them is the difference between an embedding bill that tracks the rate of change and one that tracks the size of the corpus.
The incremental re-index itself is a Python function that processes a single document: parse, normalize, chunk, embed the changed chunks, delete the chunks that no longer exist, and upsert the new ones. The content hash is the idempotency key. If the same event arrives twice, the second run sees the hash already indexed and does nothing.
def reindex_document(doc: dict, db) -> dict:
existing = db.get_document(doc["source_path"])
if existing and existing["content_hash"] == doc["content_hash"]:
db.touch_verified(existing["id"])
return {"status": "unchanged", "doc_id": existing["id"]}
chunks = chunk_hierarchical(doc)
embeddings = embed_batch([c.content for c in chunks])
with db.transaction():
doc_id = db.upsert_document(doc)
db.delete_chunks(doc_id, keep_indices=[c.chunk_index for c in chunks])
db.upsert_chunks(doc_id, chunks, embeddings)
return {"status": "reindexed", "doc_id": doc_id, "chunks": len(chunks)}
The touch_verified() call is the subtle part. Even when the content hasn't changed, the document was checked, and that check is a freshness signal. A document that is verified daily but never changes should not be re-embedded, but it should not be treated as stale either. Separating "verified" from "changed" is what makes the freshness score meaningful.
⚠️ Gotcha: Treat 404 and 410 responses as deletions, not failures. A page that is gone should leave the index, not linger as a ghost citation. The ingestion pipeline should mark the document as deleted and remove its chunks, and the retrieval layer should exclude deleted documents from all queries.
8. Freshness Scoring: Measuring the Thing That Actually Breaks
Freshness is not a boolean. It has four measurable dimensions, and each one fails in a different way:
- Content age: how long since the source document was last verified or updated.
- Embedding lag: the delay between a document's update and when its new embedding enters the index.
- Stale retrieval rate: the fraction of live queries returning a document whose embedding predates its most recent update.
- Coverage drift: the percentage of the corpus that has quietly drifted past its staleness threshold.
A nightly re-index creates up to a 24-hour embedding lag; streaming CDC reduces it to under 10 seconds. A composite freshness score below 85% should trigger an automated alert. The score is not a vanity metric; it's the signal that tells you whether the platform is doing its job.
The scoring function is straightforward once the metadata exists:
def freshness_score(doc: dict, now: datetime) -> float:
age_days = (now - doc["last_verified"]).total_seconds() / 86400
age_score = max(0.0, 1.0 - age_days / doc["max_age_days"])
lag_hours = (now - doc["last_embedded_at"]).total_seconds() / 3600
lag_score = max(0.0, 1.0 - lag_hours / doc["max_lag_hours"])
return round(0.6 * age_score + 0.4 * lag_score, 3)
The weights are a policy decision, not a technical one. A pricing page might weight age more heavily; an API reference might weight lag more heavily. The point is that the score is computed from stored metadata, not estimated from vibes, and the retrieval layer can use it to down-rank stale documents.
The second layer is staleness detection at the corpus level. A tool like ragfresh compares the source registry with the last accepted manifest, classifies each document as unchanged, changed-not-reindexed, deleted-still-indexed, or not-yet-indexed, and exits non-zero when staleness is detected, which makes it usable as a CI gate. The "deleted, still indexed" class is the worst, because it produces ghost citations — the model cites a document that no longer exists.
The third layer is retrieval feedback. When a user marks an answer as unhelpful, the chunks that were retrieved for that query are down-weighted in future retrievals. Over time, the system learns which chunks are unreliable, independent of their freshness score. This is not a replacement for re-indexing; it's a complement that catches the documents that are fresh but wrong.
💡 Practical note: Store the
last_verifiedtimestamp on every document, even the ones that never change. A document that hasn't been checked in six months is not necessarily stale, but it is unverified, and the freshness score should reflect that distinction.
9. Evaluation: Faithfulness, Context Recall, and the Metrics That Matter
A RAG system without evaluation is a system that can only improve by accident. The metrics that matter are the ones that separate retrieval quality from generation quality, because the fixes are different.
Faithfulness measures whether the answer is grounded in the retrieved context. A faithfulness score of 0.94 means the model is not hallucinating; a low score means it is generating claims that the context doesn't support.
Context recall measures whether the retrieved context contains all the information needed to answer the query. A low recall score means the retriever is missing relevant documents, and no amount of prompt engineering will fix it.
Context precision measures whether the relevant documents are ranked above the irrelevant ones. A low precision score means the retriever is returning noise, and the model is being distracted.
Answer relevancy measures whether the answer actually addresses the question. A high faithfulness and a low relevancy means the model is grounded but not helpful.
RAGAS is the standard framework for these metrics, and it is available in both Python and hosted evaluation services. The framework uses an LLM judge to score each metric, which means the evaluation itself has a cost and a bias. The practical approach is to run RAGAS on a curated evaluation set of 50–200 question-answer pairs, not on every query, and to treat the scores as a regression signal rather than an absolute truth.
A minimal evaluation harness:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
def evaluate_rag(questions: list[dict], pipeline) -> dict:
results = []
for q in questions:
answer = pipeline.ask(q["question"])
results.append({
"question": q["question"],
"answer": answer.text,
"contexts": [c["content"] for c in answer.chunks],
"ground_truth": q["ground_truth"],
})
scores = evaluate(
dataset=results,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
return scores
The ground_truth field is what makes context recall measurable. Without a reference answer, recall is a guess. The evaluation set should be built from real user queries, not synthetic questions, because synthetic questions are often cleaner and more specific than the queries the system actually receives.
🧠 The important part: Evaluate retrieval and generation separately. A faithfulness score of 0.95 with a context recall of 0.4 means the model is faithfully summarizing the wrong documents. The fix is in the retriever, not the prompt.
10. Permissions and Provenance: Who Is Allowed to Know What
A knowledge platform that serves multiple teams or tenants has a permissions problem that is independent of the retrieval quality. A chunk from the HR policy document should not be retrievable by a user who is not in HR, even if the query is semantically identical. A chunk from a customer's private contract should not be retrievable by another customer.
The 2026 consensus on enterprise knowledge assistants is that metadata, permissions, versions, and update timestamps matter as much as embeddings. The retrieval query must filter by the authenticated principal's access scope before the vector similarity is computed, not after. Filtering after the similarity search is a data leak waiting to happen, because the top-K results may include documents the user should not see, and the model may summarize them.
In PostgreSQL, the filter is a WHERE clause on the chunk's access scope, joined to the user's group memberships. The filter must run inside the same query as the vector search, so the vector index only considers rows the user is allowed to retrieve. With pgvector 0.8's iterative index scans, a selective filter doesn't under-recall; the index re-scans until enough rows survive the filter.
Provenance is the second requirement. Every chunk should carry a citation — the source document, the section, the version, and the timestamp — and every generated answer should cite the chunks it used. A citation without a timestamp is incomplete, because the user needs to know whether the source is current. A citation without a source URL is unverifiable, because the user cannot check the claim.
The third requirement is write access control. A self-updating platform where agents can write to the knowledge base is more powerful and more dangerous. The 2026 enterprise pattern is to separate read tools from write tools: the agent can read autonomously, but writes require permission checks, and high-risk writes require human approval. An agent that can silently update the knowledge base can poison it, and a poisoned knowledge base is worse than a stale one.
🚨 Production warning: Do not store the embedding model version only in application config. Store it on every chunk. When you upgrade the embedding model, the old and new vectors are not comparable, and mixing them in the same index produces silent quality degradation. The
embedding_modelcolumn is what lets you filter during the migration window.
The Build Order: What to Ship First and What to Skip
A self-updating knowledge platform is a deep system, and it is easy to spend three months on infrastructure before answering a single useful question. Here is the order I'd build in, and the things I'd deliberately leave out of v1.
Build first:
- Layout-aware ingestion for one document type. Pick the format that most of your knowledge base uses. Get the parser to separate body text from navigation chrome before you add a second format.
- Semantic chunking with parent context. Store the chunk, the context prefix, and the parent section ID. This is the single highest-leverage decision in the pipeline.
- The
chunkstable withhalfvecand a GIN index. One table, one HNSW index, one full-text index. No separate vector database. - Hybrid retrieval with RRF. Dense plus lexical, fused. This is the fix for exact identifiers, and it is cheaper than a reranker.
- A webhook-triggered incremental re-index. The webhook is five lines of n8n. The content hash is the idempotency gate. This is what makes the platform self-updating rather than self-staling.
Skip in v1:
- The full agentic loop. Start with a single retrieval and a faithfulness check. Add CRAG or Self-RAG when you have evaluation data showing that the simple pipeline fails on a meaningful fraction of queries.
- GraphRAG. A knowledge graph is valuable when the queries are relational and multi-hop. If your queries are mostly factual lookups, the graph is overhead.
- A dedicated reranker. Hybrid retrieval gets you most of the way. Add the cross-encoder when context precision is the bottleneck, not before.
- A separate vector database. pgvector with HNSW is sufficient up to roughly 10 million vectors, and the operational simplicity of one database is worth more than the marginal recall gain from a specialized engine.
The platform is not the stack. It's the loop. A system that ingests continuously, retrieves hybridly, checks its own confidence, and measures its freshness is more useful than a system with a better embedding model and a static index. The embedding model will change. The vector database may change. The loop is what stays.