AI: What is RAG? Retrieval Augmented Generation

The fastest way to make a large language model look bad is to ask it about something it cannot know: your internal refund policy, last quarter’s incident postmortems, the current pricing rules in your billing system, or the exact behavior of a private API.

The model will often answer confidently. It may also be wrong.

Retrieval Augmented Generation, usually called RAG, is the architectural answer to that problem. Instead of asking the model to memorize your private or fast-changing knowledge, you retrieve relevant information from your own data sources at query time and give that information to the model as grounded context.

But production RAG is not just “put documents into a vector database and chat with them.” The difference between a demo and a reliable system is usually found in chunking, metadata filtering, hybrid search, evaluation, permissions, and context assembly.

TL;DR: RAG is a pattern where an AI system retrieves relevant external data before generating an answer. It is useful when you need private data, fresh data, citations, auditability, or access control. It is not a magic fix for bad data, unclear questions, or unsafe prompts. A serious RAG system is part search engine, part permissions system, part prompt compiler, and part evaluation pipeline.

📋 Table of Contents

• 1. RAG Is a Grounding System, Not Just Chat With Your Docs
• 2. The Request Path: From Raw Documents to a Grounded Answer
• 3. Chunking Decides Whether Your Retrieval Is Useful
• 4. Vector Search Alone Will Disappoint You
• 5. Metadata Filters and Permissions Are Part of Retrieval
• 6. Query Understanding Is the Front Door
• 7. Context Assembly: Do Not Stuff, Curate
• 8. Evaluate Retrieval and Generation Separately
• 9. RAG vs Fine-Tuning vs Long Context
• 10. Agentic RAG: Retrieval as a Tool, Not a Single Step
• A Practical Decision Guide

---

1. RAG Is a Grounding System, Not Just Chat With Your Docs

At its core, Retrieval Augmented Generation means the model’s answer is augmented with retrieved information from outside its training data.

A minimal RAG flow looks like this:

1. A user asks a question.
2. The system searches a knowledge base for relevant passages.
3. Those passages are inserted into the model prompt.
4. The model generates an answer conditioned on the question and the retrieved context.
5. Ideally, the answer includes citations or source references.

That sounds simple, but the important mental model is this: RAG is a grounding system.

The language model is not the source of truth. Your retrieved data is. The model’s job is to reason over, summarize, compare, and explain the retrieved evidence. If retrieval returns the wrong evidence, even a strong model can produce a polished but incorrect answer.

RAG is especially useful when:

• The knowledge is private or proprietary.
• The knowledge changes frequently.
• You need citations or traceability.
• You need permission-aware answers.
• You want to avoid retraining or fine-tuning for every data update.
• You need to combine reasoning with up-to-date lookup.

RAG is not especially useful when:

• The data is tiny and can safely fit in the prompt.
• The problem is mostly style or format adaptation.
• The source data is low quality or contradictory.
• You need guaranteed deterministic answers.
• The user’s question cannot be mapped to retrievable evidence.

A lot of RAG disappointment comes from treating it as a universal fix. It is not. It is a powerful architecture for grounded generation, but only if the retrieval side is treated as a first-class product.

---

2. The Request Path: From Raw Documents to a Grounded Answer

A RAG system has two main lifecycles: ingestion and query time.

Ingestion

Ingestion is where your raw data becomes searchable knowledge.

Sources → Parsing → Cleaning → Chunking → Embedding → Indexing

Typical sources include:

• Markdown or HTML docs
• PDFs
• Support tickets
• Internal wikis
• Product catalogs
• Code repositories
• CRM notes
• Policy documents
• Meeting transcripts
• Structured database rows

Each source type has its own pain. PDFs can lose tables and layout. Wikis can have stale pages. Tickets can contain sensitive customer data. Code can be hard to chunk meaningfully. Ingestion is not glamorous, but it determines the ceiling of your retrieval quality.

Query time

Query time is where the user question becomes a grounded answer.

User question
  → Query rewriting or expansion
  → Retrieval
  → Filtering and reranking
  → Context assembly
  → Generation
  → Citation and evaluation

A simple local example with embeddings helps show the core mechanic:

# pip install sentence-transformers numpy
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

chunks = [
    "Refunds are approved within 14 days of purchase.",
    "Enterprise SLAs include 24/7 support.",
    "Passwords must be rotated every 90 days.",
]

embeddings = model.encode(chunks, normalize_embeddings=True)

query = "What is the refund window?"
query_embedding = model.encode(query, normalize_embeddings=True)

scores = embeddings @ query_embedding

for idx in np.argsort(scores)[::-1]:
    print(scores[idx], chunks[idx])

This is not a full production system, but it shows the basic idea: the question and the documents are mapped into vectors, and the closest chunks become candidates for the model prompt.

🔍 Why this matters: The model does not magically “know” your documents. It only sees what you put into the prompt. If the retrieval step misses the right passage, the generation step cannot recover from that absence.

---

3. Chunking Decides Whether Your Retrieval Is Useful

Chunking is the process of splitting documents into retrievable units. It is one of the most underestimated parts of RAG.

Scenario: Your company has a support policy that says refunds are allowed within 14 days, except for enterprise contracts, which follow a separate addendum. If your chunker cuts the policy in the wrong place, the retrieved chunk may say “refunds are allowed within 14 days” but omit the enterprise exception. The model then gives a confidently wrong answer.

The obvious solution is to chunk by fixed token length. That works for prototypes, but it often breaks semantic boundaries. A paragraph about pricing may be split from the paragraph that defines its exceptions. A table may be chopped into meaningless rows. A code function may be split from its docstring.

Better chunking strategies include:

• Fixed-size chunks with overlap — simple and predictable, but semantically blunt.
• Structure-aware chunking — use headings, sections, paragraphs, tables, and code boundaries.
• Parent-child chunking — retrieve small precise chunks but pass larger parent sections to the model.
• Semantic chunking — split based on embedding similarity or topic shifts.
• Document-type-specific chunking — treat legal docs, code, tickets, and product catalogs differently.

A practical Markdown chunker might look like this:

def chunk_markdown_by_heading(markdown: str, max_chars: int = 1200) -> list[dict]:
    chunks = []
    heading = ""
    buffer = []

    def flush():
        if not buffer:
            return

        text = "\n".join(buffer).strip()
        if not text:
            return

        if len(text) <= max_chars:
            chunks.append({"heading": heading, "text": text})
            return

        step = max_chars - 200
        for start in range(0, len(text), step):
            part = text[start:start + max_chars]
            if part.strip():
                chunks.append({"heading": heading, "text": part})

    for line in markdown.splitlines():
        if line.startswith("#"):
            flush()
            heading = line.lstrip("#").strip()
            buffer = [line]
        else:
            buffer.append(line)

    flush()
    return chunks

This is still simple, but it respects headings better than slicing the whole document blindly.

A few chunking rules I would apply in most projects:

• Preserve section titles with each chunk.
• Keep tables intact when possible.
• Add source URL or document ID to every chunk.
• Avoid chunks so small that they lack context.
• Avoid chunks so large that they dilute relevance.
• Test chunking with real user questions, not just visually.

⚠️ Gotcha: If your answers depend on exceptions, conditions, or tables, naive chunking will hurt you first. The model will usually sound fine while being subtly wrong.

---

4. Vector Search Alone Will Disappoint You

Vector search is powerful, but it is not a complete search engine.

Embeddings are good at capturing semantic similarity. They can match “How do I reset my password?” with “Credential rotation procedure.” That is useful. But vector search can struggle with exact identifiers, error codes, product names, part numbers, rare terms, and precise keyword matches.

If a user asks for ERR_PAYMENT_GATEWAY_TIMEOUT, a pure embedding search may retrieve generic payment failure documents instead of the exact error page. Keyword search can find the exact token more reliably.

That is why serious RAG systems often use hybrid retrieval:

• Vector search for semantic matching.
• Keyword search for exact terms and lexical matching.
• Metadata filters for scope and permissions.
• A reranker to order the final candidates.

A PostgreSQL example using pgvector and full-text search can illustrate the idea. This query uses Reciprocal Rank Fusion to combine vector and keyword results without assuming their raw scores are comparable:

WITH vector_hits AS (
  SELECT
    id,
    ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rank
  FROM documents
  WHERE tenant_id = $3
  ORDER BY embedding <=> $1::vector
  LIMIT 100
),

text_hits AS (
  SELECT
    id,
    ROW_NUMBER() OVER (
      ORDER BY ts_rank_cd(tsv, websearch_to_tsquery('english', $2)) DESC
    ) AS rank
  FROM documents
  WHERE tenant_id = $3
    AND tsv @@ websearch_to_tsquery('english', $2)
  LIMIT 100
),

fused AS (
  SELECT
    id,
    SUM(1.0 / (60 + rank)) AS score
  FROM (
    SELECT id, rank FROM vector_hits
    UNION ALL
    SELECT id, rank FROM text_hits
  ) combined
  GROUP BY id
)

SELECT
  d.id,
  d.content,
  d.source_url,
  f.score
FROM fused f
JOIN documents d ON d.id = f.id
ORDER BY f.score DESC
LIMIT 20;

The exact constants are not sacred. The point is the shape: retrieve broadly, combine signals, then rerank.

After hybrid retrieval, many systems pass the top 20, 50, or 100 candidates to a reranker. A reranker can be a cross-encoder model or a hosted reranking service. Its job is to compare the query and passage more carefully than the initial embedding search can.

A common pattern:

Query → retrieve 100 candidates → rerank → keep top 5 to 10 → generate

This is usually better than sending the top 100 chunks straight to the model. More context is not automatically better context.

💡 Practical note: If your users search for error codes, SKUs, ticket IDs, contract names, or exact API fields, do not rely on vector similarity alone. Hybrid search is not optional in those domains.

---

5. Metadata Filters and Permissions Are Part of Retrieval

A RAG system that retrieves the most semantically similar document may still be wrong because the user is not allowed to see it.

Scenario: An employee asks, “What is the compensation band for senior engineers?” The best-matching document is an HR spreadsheet, but the employee is not in HR. If your retriever returns it, your AI assistant just became a permission leak.

This is why metadata is not a decorative feature. Every chunk should carry enough metadata to answer questions like:

• Which tenant does this belong to?
• Which user roles can access it?
• Is it public, internal, or confidential?
• What product version does it apply to?
• Is it archived or current?
• What is its source system?
• When was it last updated?

A document index might store chunks like this:

{
  "id": "doc_987_chunk_4",
  "tenant_id": "acme",
  "source_url": "https://wiki.acme.example/refund-policy",
  "department": "support",
  "access_roles": ["support_agent", "support_manager"],
  "product_version": "2026.1",
  "status": "active",
  "updated_at": "2026-01-14",
  "text": "Refunds are approved within 14 days of purchase..."
}

At query time, the retriever should apply filters before returning results:

filters = {
    "tenant_id": current_user.tenant_id,
    "status": "active",
    "access_roles": {"$in": current_user.roles},
}

results = vector_store.search(
    query_embedding=query_embedding,
    filters=filters,
    limit=50,
)

The exact filter syntax depends on your database or vector store, but the architectural rule is the same: authorization must happen during retrieval, not after generation.

Do not rely on the model to “avoid mentioning” restricted information. If sensitive content enters the prompt, it can influence the answer, leak partially, or be extracted through prompt manipulation.

Also beware of post-filtering. If you retrieve the top 100 vector matches and then filter out unauthorized documents, you may end up with too few relevant results. It is usually better to push filters into the search engine itself.

🚨 Production warning: Multi-tenant RAG systems need strict tenant isolation. A missing tenant_id filter is one of the easiest and most dangerous bugs to ship.

---

6. Query Understanding Is the Front Door

Users rarely ask perfect retrieval questions.

They ask things like:

• “Why is billing broken?”
• “What did we decide about the cache thing?”
• “Can customers use SSO on the old plan?”
• “The export failed again.”

These questions are underspecified. A RAG system often needs a query-understanding layer before retrieval.

Common techniques include:

• Query rewriting: Turn a vague user question into a clearer search query.
• Query expansion: Generate multiple candidate queries to improve recall.
• Question decomposition: Break a multi-part question into subquestions.
• Entity extraction: Identify product names, error codes, versions, or customer IDs.
• Intent classification: Decide whether retrieval is even needed.
• Clarifying questions: Ask the user for missing scope when the risk of guessing is high.

A simple query expansion prompt might look like this:

def expand_query(question: str) -> list[str]:
    prompt = f"""
    Generate 3 search queries for a knowledge base.
    Do not answer the question.
    Preserve exact identifiers when present.
    Question: {question}
    """
    return llm.generate_list(prompt)

For example, “SSO on the old plan” might become:

1. Single sign-on support for legacy pricing plans
2. SSO availability old plan enterprise authentication
3. Legacy plan SAML OIDC single sign-on limitations

This can improve recall, but it also introduces risk. Query rewriting can accidentally change the meaning. If the user asks about version 2 and the rewriter drops the version constraint, retrieval may return version 3 documentation.

Use query transformation carefully, especially in domains where exact names, versions, and identifiers matter.

A good rule:

• For casual internal search, aggressive rewriting may be fine.
• For compliance, legal, medical, billing, or engineering diagnostics, preserve exact terms and be conservative.

The front door of a RAG system is not the vector database. It is the interpretation of the user’s question.

---

7. Context Assembly: Do Not Stuff, Curate

Once retrieval returns candidates, you need to turn them into a prompt. This step is more important than many teams expect.

The naive approach is to stuff as many chunks as possible into the context window. That often degrades quality. The model may receive contradictory passages, outdated information, duplicated content, or weakly relevant material. The final answer can become vague, hedged, or polluted by unrelated details.

A better context assembler does several things:

• Removes duplicates and near-duplicates.
• Orders passages by relevance.
• Keeps the most important evidence visible.
• Includes source identifiers for citations.
• Truncates responsibly.
• Respects a token budget.
• Separates retrieved evidence from user instructions.

Example:

def build_prompt(question: str, passages: list[dict], max_sources: int = 6) -> str:
    context_blocks = []

    for i, passage in enumerate(passages[:max_sources], start=1):
        context_blocks.append(
            f"[{i}] {passage['text']}\nSource: {passage['source_url']}"
        )

    context = "\n\n".join(context_blocks)

    return f"""
Answer the user's question using only the numbered sources below.
If the sources do not contain enough information, say that clearly.
Cite sources using bracket numbers like [1] and [3].

Sources:
{context}

Question:
{question}
""".strip()

This prompt is intentionally restrictive. It tells the model to use the sources, cite them, and admit when the evidence is insufficient.

In production, you may also want to add:

• A system message defining tone and safety rules.
• Metadata such as document date or product version.
• Instructions for handling conflicting sources.
• A rule to prefer newer documents when dates conflict.
• A rule not to use hidden or internal metadata in the final answer.

Long context windows are useful, but they do not remove the need for curation. Even when a model can accept many tokens, irrelevant context increases cost, latency, and the chance of distraction.

The prompt is not just text. It is the runtime memory manager for the generation step.

---

8. Evaluate Retrieval and Generation Separately

If your RAG system gives a bad answer, you need to know why.

Was the right document missing from the index?
Was it chunked badly?
Did retrieval miss it?
Did retrieval find it but rank it too low?
Was the right passage filtered out by permissions?
Was the prompt poorly assembled?
Did the model ignore the context?
Did the model hallucinate despite good context?

To answer these questions, evaluate retrieval and generation separately.

Retrieval metrics

Retrieval evaluation asks: Did we find the right evidence?

Useful metrics include:

• Recall@k: Are the correct passages in the top k results?
• Precision@k: How many top results are actually relevant?
• MRR: How high is the first relevant result ranked?
• nDCG: Are more relevant results ranked higher?
• Filter correctness: Are unauthorized or outdated documents excluded?

A minimal evaluation case might look like this:

evaluation_cases = [
    {
        "question": "What is the standard refund window?",
        "expected_doc_ids": ["refund_policy_v3#section-2"],
        "forbidden_doc_ids": ["refund_policy_2021_archived"],
    },
    {
        "question": "Does the legacy plan support SSO?",
        "expected_doc_ids": ["legacy_plan_features#sso"],
        "forbidden_doc_ids": [],
    },
]

Generation metrics

Generation evaluation asks: Given the retrieved evidence, did the model produce a faithful answer?

Useful checks include:

• Faithfulness: Is the answer supported by the retrieved context?
• Answer relevance: Does it actually answer the question?
• Citation accuracy: Do the citations point to supporting passages?
• Refusal quality: Does the system say “I don’t know” when evidence is missing?
• Policy compliance: Does it avoid unsafe or prohibited claims?

A simple automated check can ask another model to judge faithfulness, but for high-risk systems you should also use human review and domain-specific test suites.

The key is to avoid evaluating the whole system only by vibes. If you cannot separate retrieval failures from generation failures, you will tune the wrong part of the pipeline.

🧠 The important part: A RAG system is only as debuggable as its trace. Log the query, rewritten queries, retrieved chunks, reranked chunks, final prompt, model output, and citations for every important request.

---

9. RAG vs Fine-Tuning vs Long Context

RAG is not the only way to give an AI system domain knowledge. The main alternatives are fine-tuning and long-context prompting.

Choosing between them is an architectural decision.

Approach	Best for	Strengths	Weaknesses
RAG	Private, changing, citation-sensitive data	Freshness, auditability, access control, easier updates	Requires retrieval infrastructure and evaluation
Fine-tuning	Style, format, specialized task behavior	Can improve consistency and task performance	Expensive to update; poor for frequently changing facts
Long-context prompting	Small corpora, prototypes, simple document Q&A	Simple to start	Cost, latency, permissions, and context dilution issues
Hybrid approach	Real products	Combines strengths	More complexity

A useful way to think about it:

• Use RAG when the model needs to look up facts.
• Use fine-tuning when the model needs to behave differently.
• Use long context when the relevant data is small enough to include directly and you do not need strict retrieval controls.

For example, if you want support answers to follow a specific tone, use a system prompt or fine-tuning. If you want the model to answer based on yesterday’s policy update, use RAG. If you want the model to answer from a three-page contract that the user uploaded, putting the whole contract into context may be perfectly reasonable.

The mistake is using fine-tuning to teach the model facts that change every week. That creates a stale model and a painful update process.

Another mistake is using long-context prompting as a permanent substitute for retrieval. It can work for small documents, but it becomes harder when you need permissions, freshness, citations, cost control, and large knowledge bases.

In many production systems, the best answer is hybrid:

• RAG for knowledge lookup.
• System prompts for behavior and policy.
• Fine-tuning only when a clear task-level benefit is measured.
• Long context for selected documents or final-mile reasoning.

---

10. Agentic RAG: Retrieval as a Tool, Not a Single Step

Basic RAG performs one retrieval step and then generates. Agentic RAG treats retrieval as a tool that the model can use repeatedly.

Instead of:

Question → retrieve once → answer

You get:

Question → retrieve → inspect → refine query → retrieve again → compare sources → answer

This is useful when questions require iteration:

• “Compare the 2025 and 2026 enterprise SLAs.”
• “Find the root cause of the payment outage and summarize the fix.”
• “Which policy applies to this customer contract?”
• “What changed between release 4.1 and 4.2?”

An agent may need to retrieve from multiple sources: documentation, ticket history, code, structured databases, and internal policies. It may need to check a date, then fetch a related document, then verify a contradiction.

A simple tool schema might look like this:

{
  "name": "search_knowledge_base",
  "description": "Search internal documents, policies, and support articles.",
  "parameters": {
    "query": {
      "type": "string"
    },
    "filters": {
      "type": "object",
      "properties": {
        "product": { "type": "string" },
        "department": { "type": "string" },
        "updated_after": { "type": "string" }
      }
    },
    "limit": {
      "type": "integer"
    }
  }
}

Agentic RAG can be much more capable than single-shot RAG, but it also introduces new failure modes:

• Infinite retrieval loops.
• Excessive cost and latency.
• Inconsistent tool usage.
• Overconfidence after partial evidence.
• Prompt injection through retrieved content.
• Difficulty reproducing answers.

Guardrails matter.

In production, I would add:

• Maximum number of retrieval steps.
• Timeout and token budget.
• Caching for repeated queries.
• Logging of every tool call.
• Restrictions on which tools can be combined.
• A final verification step for high-risk answers.
• Human escalation when evidence is weak or contradictory.

Agentic RAG is not automatically better. It is more flexible, but also harder to operate. Start with a simple RAG pipeline. Add agency when real user questions show that one retrieval step is not enough.

---

A Practical Decision Guide

If I were deciding whether to use RAG, I would ask these questions in order.

Use RAG when you need:

• Answers grounded in private or external data.
• Fresh knowledge without retraining.
• Citations or source links.
• Permission-aware retrieval.
• Auditability for compliance or support.
• The ability to update knowledge independently of the model.

Avoid RAG when:

• The relevant context is tiny and can be safely included directly.
• The problem is mostly tone, formatting, or task style.
• Your data is unreliable, duplicated, or unowned.
• You cannot define what a correct answer looks like.
• You need deterministic behavior with no retrieval uncertainty.
• The workflow is better served by ordinary search or a database query.

A sensible starting architecture

For most production RAG systems, I would start with:

1. A golden evaluation set of realistic questions and expected sources.
2. Structure-aware ingestion that preserves headings, tables, and metadata.
3. Hybrid retrieval using vector and keyword search.
4. Strict metadata filtering for tenant, role, status, and version.
5. A reranking step before prompt assembly.
6. A disciplined prompt with citations and refusal instructions.
7. Logging and tracing for every retrieval and generation step.
8. A feedback loop where bad answers become new test cases.

The most important insight is that RAG is not a single model feature. It is a system design choice. The language model is only one component. The real reliability comes from how well you ingest, filter, retrieve, rank, evaluate, and ground the information before the first token is generated.

A good RAG system does not make the model omniscient. It makes the model accountable.