How to Reduce RAG Costs with Embedding Caches, Top-K Tuning, and Context Compression

The first RAG bill usually arrives after the demo phase.

The prototype looked cheap: a few hundred questions, a small document set, and a vector database running somewhere modest. Then production happens. Users ask long questions. Documents get re-indexed. The retriever pulls ten chunks by default. Each chunk is 800 tokens. The model receives 8,000 tokens of context before it writes a single word. Suddenly the biggest cost is not “AI magic.” It is input tokens.

Retrieval-Augmented Generation costs tend to hide in three places:

• repeated embedding work,
• retrieving more context than the answer needs,
• stuffing that context into the prompt without compression or selection.

The good news is that these are engineering problems, not mystical model problems. You can reduce RAG costs significantly with embedding caches, top-K tuning, and context compression. But each technique has a failure mode. Cache invalidation can make embeddings stale. Aggressive top-K cuts can destroy recall. Compression can remove the exact sentence the answer depended on.

This article looks at the practical versions of these techniques: what to cache, how to tune retrieval depth, when to compress, and how to avoid quietly degrading answer quality.

TL;DR

• RAG cost is usually dominated by LLM input tokens, not embedding generation.
• Exact embedding caches are safe and useful when keyed by model, text, and schema version.
• Query caches help when queries repeat; semantic caches help when intent repeats, but need strict scoping.
• Do not use a fixed top-K if your score distribution varies. Use a policy.
• Retrieving more candidates and reranking to fewer final chunks often beats blindly lowering K.
• Prefer extractive context compression over free-form summarization for factual RAG.
• Always measure cost per acceptable answer, not just cost per request.

📋 Table of Contents

• The cost model most teams miss
• 1. Exact embedding caches: stop paying for the same chunk twice
• 2. Query embedding caches only pay off when queries repeat
• 3. Semantic caching can reduce repeated intent, but it needs guardrails
• 4. Top-K is not a constant; it is a policy
• 5. Retrieve more, prompt less: the rerank pattern
• 6. Context compression: remove tokens, not evidence
• 7. Chunking and parent-child retrieval are hidden cost controls
• 8. Metadata filters and query routing shrink the search space
• 9. Evaluate before squeezing: cost cuts can quietly destroy quality
• Choosing the right combination

The cost model most teams miss

A useful way to think about RAG cost is:

Total RAG cost ≈
  ingestion embedding cost
+ query embedding cost
+ vector storage and search cost
+ reranking cost
+ LLM input token cost
+ LLM output token cost
+ evaluation and observability cost

Teams often focus on embedding pricing first because embeddings are visible and easy to count. But in many production systems, the recurring cost is the prompt. If every answer sends eight retrieved chunks, each with 600 tokens, plus system instructions and conversation history, the LLM input can dwarf the embedding cost.

That does not mean embedding caches are unimportant. They are useful, especially during ingestion, re-indexing, and repeated query workloads. But they are only one lever.

The three levers in this article attack different parts of the cost curve:

Lever	Main cost reduced	Main risk
Embedding caches	Repeated embedding calls	Stale vectors, bad invalidation
Top-K tuning	Retrieved context size	Lost recall
Context compression	Prompt tokens	Removed evidence, hallucinated summaries

The goal is not “spend as little as possible.” The goal is to spend less while preserving groundedness, recall, and answer usefulness.

1. Exact embedding caches: stop paying for the same chunk twice

Scenario:
Your document pipeline reprocesses the same policy PDFs every night. Most chunks are unchanged, but the pipeline re-embeds everything anyway. Meanwhile, boilerplate sections like “Contact support” and “Refund eligibility” appear in dozens of documents.

Why it matters:
Embedding generation is often cheap per call, but repeated ingestion at scale adds up. More importantly, re-embedding unchanged content wastes time, rate limits, and pipeline capacity. If you are embedding millions of chunks, exact caching can make re-indexing dramatically cheaper.

Solution:
Use a content-addressed embedding cache. The cache key should include:

• embedding model name,
• embedding model or schema version,
• normalization version,
• the exact text being embedded.

Do not key only on the document ID. The same document can have different chunk text after splitting changes. Do not key only on text either, because changing the embedding model makes old vectors incompatible.

A practical Python example:

import hashlib
import json
from typing import List, Protocol


class EmbeddingClient(Protocol):
    def embed(self, model: str, texts: List[str]) -> List[List[float]]:
        ...


def embedding_cache_key(model: str, text: str, cache_version: str = "v1") -> str:
    normalized_text = " ".join(text.strip().split())

    payload = json.dumps(
        {
            "model": model,
            "cache_version": cache_version,
            "text": normalized_text,
        },
        sort_keys=True,
        ensure_ascii=False,
    )

    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
    return f"emb:{model}:{cache_version}:{digest}"


def get_embeddings(
    client: EmbeddingClient,
    cache,
    model: str,
    texts: List[str],
    ttl_seconds: int | None = None,
) -> List[List[float]]:
    results: List[List[float] | None] = [None] * len(texts)
    missing_indices: List[int] = []
    keys: List[str] = []

    for i, text in enumerate(texts):
        key = embedding_cache_key(model, text)
        keys.append(key)

        cached = cache.get(key)

        if cached is not None:
            results[i] = json.loads(cached)
        else:
            missing_indices.append(i)

    if missing_indices:
        missing_texts = [texts[i] for i in missing_indices]
        missing_vectors = client.embed(model, missing_texts)

        for idx, vector in zip(missing_indices, missing_vectors):
            results[idx] = vector
            cache.set(
                keys[idx],
                json.dumps(vector),
                ex=ttl_seconds,
            )

    return results

This example assumes a Redis-like cache with get and set. In production, you may want a more compact serialization format than JSON, such as MessagePack or raw bytes, especially for high-dimensional vectors.

Why this works:
The cache is deterministic. If the text, model, and cache version are unchanged, the embedding should be reusable. If the model changes, the key changes. If you change normalization rules, you can bump cache_version.

💡 Practical note:
Include chunking schema versioning if chunk text can change due to splitter updates. A chunk that looks “the same” to a human may not be byte-for-byte identical after a pipeline change.

Exact embedding caches are especially valuable for:

• nightly re-indexing jobs,
• documents with shared boilerplate,
• multilingual corpora with repeated sections,
• test and staging environments,
• regression evaluation runs.

They are less magical for unique user queries, which brings us to the next point.

2. Query embedding caches only pay off when queries repeat

Scenario:
Your support assistant receives many variations of “Where is my invoice?” and “How do I reset my password?” Each question still triggers query embedding, retrieval, and generation.

Why it matters:
If the same query repeats often, caching the query embedding can save embedding calls. But the bigger win is often caching the downstream result: the retrieved context or the final answer. Query embedding alone may be a small cost compared with LLM generation.

Solution:
Use an exact query cache for normalized queries, but be careful about what “same query” means.

A safe exact query cache key should include:

• normalized query text,
• tenant or workspace ID,
• user access scope if retrieval is permission-aware,
• retrieval configuration version,
• embedding model version.

Example:

def query_cache_key(
    query: str,
    tenant_id: str,
    access_scope: str,
    retrieval_config_version: str,
    model: str,
) -> str:
    normalized_query = " ".join(query.strip().lower().split())

    payload = json.dumps(
        {
            "type": "rag_query_cache",
            "tenant_id": tenant_id,
            "access_scope": access_scope,
            "retrieval_config_version": retrieval_config_version,
            "model": model,
            "query": normalized_query,
        },
        sort_keys=True,
        ensure_ascii=False,
    )

    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()
    return f"rag:query:{tenant_id}:{digest}"

This key is intentionally conservative. It does not assume that two users in different tenants can share results. It also does not assume that a retrieval configuration change can reuse old results.

Why this works:
RAG responses are often context-dependent. If the user’s permissions, tenant data, or retrieval settings change, the cached answer may be wrong. Including those factors in the cache key prevents accidental cross-tenant or cross-scope leakage.

When exact query caching helps

• FAQ-style assistants
• public documentation search
• internal tools with repeated operational questions
• evaluation harnesses that replay the same queries
• support bots with common troubleshooting flows

When exact query caching does not help much

• highly personalized queries
• long conversational threads where the query includes history
• analytics questions with unique filters or date ranges
• queries over rapidly changing data
• low-repeat traffic patterns

⚠️ Gotcha:
Do not cache answers for queries whose correctness depends on fresh data unless you have a very short TTL or an invalidation strategy. “What is my order status?” is not the same kind of query as “What is your refund policy?”

3. Semantic caching can reduce repeated intent, but it needs guardrails

Exact caching misses paraphrases. A user asks:

How can I change my billing address?

Another asks:

Where do I update the address on my invoices?

These are not identical strings, but they may have the same intent.

Semantic caching tries to solve this by storing query embeddings and checking whether a new query is close enough to a previous one. If it is, the system can reuse the previous retrieval result or final answer.

This can reduce cost, but it is also where teams get burned.

Scenario:
A customer asks, “Can I cancel my subscription?” Later another customer asks, “Can I cancel my order?” The embeddings are close. The cache returns the wrong answer.

Why it matters:
Semantic similarity is not semantic equivalence. Two questions can be near each other in embedding space but require different business logic, permissions, or factual answers.

Solution:
Use semantic caching only when the answer is stable, low-risk, and not user-specific. Then add guardrails:

• high similarity threshold,
• tenant or namespace isolation,
• short TTL,
• explicit cache scope,
• fallback to normal RAG when uncertain,
• monitoring for cache miss/hit quality.

A simplified semantic cache lookup might look like this:

import numpy as np


def cosine_similarity(a: List[float], b: List[float]) -> float:
    a_arr = np.asarray(a)
    b_arr = np.asarray(b)

    denominator = np.linalg.norm(a_arr) * np.linalg.norm(b_arr)

    if denominator == 0:
        return 0.0

    return float(np.dot(a_arr, b_arr) / denominator)


def find_semantic_cache_hit(
    query_embedding: List[float],
    cached_entries: List[dict],
    threshold: float = 0.92,
) -> dict | None:
    best_entry = None
    best_score = -1.0

    for entry in cached_entries:
        score = cosine_similarity(query_embedding, entry["embedding"])

        if score > best_score:
            best_score = score
            best_entry = entry

    if best_entry is not None and best_score >= threshold:
        return best_entry

    return None

In production, you would not scan all cached entries in Python. You would use a vector index, Redis vector search, pgvector, Qdrant, or another store. But the logic is the same: find the closest cached query embedding, then decide whether the similarity is high enough to trust.

What to cache semantically

Good candidates:

• public FAQ answers,
• documentation explanations,
• stable policy questions,
• error message explanations,
• internal troubleshooting playbooks.

Bad candidates:

• account-specific questions,
• real-time inventory or order status,
• permissions-sensitive answers,
• legal or compliance advice,
• anything where a small wording difference changes the meaning.

🚨 Production warning:
Semantic caches must be scoped by tenant and access policy. A high similarity score does not mean two users are allowed to see the same answer.

Semantic caching can reduce cost, but it should be treated as an optimization layer, not the foundation of your RAG architecture.

4. Top-K is not a constant; it is a policy

Scenario:
Your retriever is configured with top_k = 10 because that was reasonable during prototyping. Each chunk is 700 tokens. The prompt now includes 7,000 tokens of retrieved context, even when the answer only needed one paragraph.

Why it matters:
Fixed top-K ignores three things:

1. retrieval confidence,
2. chunk length,
3. the actual information needs of the question.

A simple factual question may need one or two chunks. A comparative question may need several. A low-confidence query may need more context or no answer at all.

Solution:
Replace fixed top-K with a selection policy.

A good policy considers:

• the top retrieval score,
• the gap between top scores,
• a minimum score threshold,
• a maximum token budget,
• the maximum number of chunks allowed.

Here is a simple score-aware selector:

from typing import List


class RetrievalHit:
    def __init__(self, text: str, score: float, source: str):
        self.text = text
        self.score = score
        self.source = source


def choose_top_k(
    hits: List[RetrievalHit],
    min_k: int = 1,
    max_k: int = 6,
    min_score: float = 0.25,
    relative_drop: float = 0.20,
) -> List[RetrievalHit]:
    if not hits:
        return []

    hits = sorted(hits, key=lambda hit: hit.score, reverse=True)

    top_score = hits[0].score

    if top_score < min_score:
        return []

    selected = [hits[0]]

    for hit in hits[1:]:
        if len(selected) >= max_k:
            break

        if hit.score < min_score:
            break

        if len(selected) >= min_k and hit.score < top_score * (1 - relative_drop):
            break

        selected.append(hit)

    return selected

This logic says:

• always consider the best hit if it passes the threshold,
• stop when scores drop too far from the top hit,
• stop when the maximum chunk count is reached,
• return nothing if the best hit is too weak.

The exact threshold depends on your embedding model, normalization strategy, and score type. If your vector database returns distances instead of similarities, invert the logic.

Token-aware selection

Chunk count is a weak proxy for prompt cost. Ten short chunks may be cheaper than three long ones.

A more production-oriented selector checks token budget:

def select_within_token_budget(
    hits: List[RetrievalHit],
    count_tokens,
    token_budget: int,
    max_k: int = 6,
) -> List[RetrievalHit]:
    selected = []
    used_tokens = 0

    for hit in hits:
        if len(selected) >= max_k:
            break

        hit_tokens = count_tokens(hit.text)

        if used_tokens + hit_tokens > token_budget:
            continue

        selected.append(hit)
        used_tokens += hit_tokens

    return selected

The count_tokens function should use the tokenizer associated with your LLM where possible. Character-count heuristics are okay for rough estimates, but they can mislead when documents contain code, tables, or non-English text.

Why this works:
A policy-based retriever adapts to the query. Easy queries get less context. Ambiguous queries can get more, within a budget. Low-confidence queries can trigger clarification or fallback behavior instead of forcing an answer from weak evidence.

🔍 Why this matters:
Lowering top-K is not automatically a cost optimization. If lowering K causes retries, follow-up questions, or bad answers, you may increase total system cost.

5. Retrieve more, prompt less: the rerank pattern

A common mistake is to assume that the embedding model’s top five results should go directly into the prompt.

Sometimes that works. But embedding models are usually optimized for broad similarity search, not for the final ranking decision. They may retrieve a plausible set of chunks, but the most useful chunk might be at position 9 or 12.

This creates a trade-off:

• If you retrieve only five chunks, you may miss the right evidence.
• If you put fifteen chunks into the prompt, you pay for too many tokens.

The rerank pattern solves this by separating candidate retrieval from final selection.

Scenario:
Your vector search retrieves 30 chunks. A cross-encoder or reranking model scores each chunk against the query more precisely. You then keep only the top 3 or 4 chunks for the LLM.

Why it matters:
Reranking can improve answer quality while reducing prompt size. You retrieve broadly, then prompt narrowly.

A typical architecture:

Query
  → vector search top 30 or top 50
  → metadata filtering
  → reranker scores query-chunk pairs
  → keep top 3 to 5
  → optional compression
  → LLM prompt

The reranker can be:

• a cross-encoder model,
• a small LLM used for relevance scoring,
• a keyword/BM25 hybrid scorer,
• a domain-specific ranking model.

The important part is that the reranker sees the query and candidate chunk together. That is usually more precise than comparing their embeddings independently.

Cost trade-off

Reranking adds compute. It may also add latency. But it can reduce LLM input tokens and improve answer quality at the same time.

This is one of the few optimizations that can make the system both better and cheaper, but only if the original retrieval was over-fetching.

Pattern	Retrieval cost	Prompt cost	Answer quality risk
Fixed top-5 embedding only	Low	Medium	May miss relevant chunks
Fixed top-12 embedding only	Medium	High	More context, more noise
Retrieve 30, rerank to 4	Higher retrieval/rerank	Lower prompt	Usually better if reranker is good

When reranking is worth it

• long chunks,
• noisy corpora,
• many similar documents,
• high cost per LLM call,
• quality problems caused by irrelevant context,
• retrieval recall problems at low K.

When reranking may not be worth it

• very low-latency requirements,
• tiny document sets,
• already concise chunks,
• embedding search is highly precise,
• reranker cost exceeds LLM token savings.

💡 Practical note:
Start by retrieving more candidates than you intend to prompt. If retrieving 30 and reranking to 4 improves quality while reducing prompt tokens, you have found a useful trade-off. If it only adds latency, simplify.

6. Context compression: remove tokens, not evidence

Scenario:
Your retriever returns a 1,200-token policy section. The answer depends on two sentences, but the rest of the chunk contains definitions, examples, and boilerplate. The model reads all of it. You pay for all of it.

Why it matters:
Retrieved chunks often contain more text than the model needs. Compression can reduce prompt tokens, but naive summarization can remove important constraints, dates, exceptions, or citations.

Solution:
Use context compression carefully. For factual RAG, prefer extractive compression over generative summarization.

Extractive compression selects relevant sentences or passages from the original text. Generative summarization rewrites the text. Rewriting can be useful, but it introduces more risk: the summary may omit a condition, flatten a nuance, or hallucinate a detail.

A safe compression pipeline looks like this:

Retrieved chunks
  → split into sentences or small passages
  → score each sentence against the query
  → select highest-scoring sentences within token budget
  → preserve source IDs and positions
  → assemble compressed context

Example:

from typing import Callable, List


class Chunk:
    def __init__(self, text: str, source: str, score: float):
        self.text = text
        self.source = source
        self.score = score


class CompressedSentence:
    def __init__(
        self,
        text: str,
        source: str,
        position: int,
        retrieval_score: float,
        relevance_score: float,
    ):
        self.text = text
        self.source = source
        self.position = position
        self.retrieval_score = retrieval_score
        self.relevance_score = relevance_score


def compress_context(
    query: str,
    chunks: List[Chunk],
    sentence_split: Callable[[str], List[str]],
    relevance_scorer: Callable[[str, str], float],
    count_tokens: Callable[[str], int],
    token_budget: int,
) -> List[CompressedSentence]:
    candidates: List[CompressedSentence] = []

    for chunk in chunks:
        sentences = sentence_split(chunk.text)

        for position, sentence in enumerate(sentences):
            sentence = sentence.strip()

            if not sentence:
                continue

            relevance_score = relevance_scorer(query, sentence)

            candidates.append(
                CompressedSentence(
                    text=sentence,
                    source=chunk.source,
                    position=position,
                    retrieval_score=chunk.score,
                    relevance_score=relevance_score,
                )
            )

    candidates.sort(key=lambda c: c.relevance_score, reverse=True)

    selected: List[CompressedSentence] = []
    used_tokens = 0

    for candidate in candidates:
        sentence_tokens = count_tokens(candidate.text)

        if used_tokens + sentence_tokens > token_budget:
            continue

        selected.append(candidate)
        used_tokens += sentence_tokens

    selected.sort(key=lambda c: (c.source, c.position))

    return selected

The relevance_scorer can be implemented in several ways:

• a cross-encoder model,
• a small local model,
• BM25-style lexical scoring,
• keyword overlap,
• an LLM scoring step with strict output constraints.

For production, a cross-encoder or lightweight relevance model is often a better fit than asking a large LLM to summarize everything.

What compression should preserve

Good context compression keeps:

• source identifiers,
• section titles if available,
• dates,
• numeric values,
• conditions and exceptions,
• exact names or IDs,
• enough surrounding text for grounding.

Bad compression removes:

• “unless” clauses,
• eligibility requirements,
• version numbers,
• limitations,
• negative statements,
• citation anchors.

When generative summarization is acceptable

Generative summarization can be useful when:

• the user asks for a high-level summary,
• exact wording is less important,
• the source text is long and repetitive,
• the summary is clearly labeled as generated,
• citations still link to original passages.

For compliance, legal, medical, financial, or operational answers, extractive compression is usually safer.

⚠️ Gotcha:
Compression can reduce input token cost while increasing another model cost. If your compressor is an LLM, measure the full pipeline cost and latency, not just the final prompt size.

7. Chunking and parent-child retrieval are hidden cost controls

Chunking is usually discussed as a quality problem, but it is also a cost problem.

If chunks are too large, every retrieval drags in unnecessary text. If chunks are too small, the model may lack context and require more chunks to form a correct answer.

Scenario:
You chunk documents into 2,000-token sections to preserve context. Retrieval works well, but every answer includes massive chunks. Prompt costs rise. You switch to 200-token chunks. Costs fall, but answers become fragmented because the model misses the surrounding section.

Why it matters:
Chunk size affects top-K, compression, reranking, and final prompt size. You cannot tune RAG cost without thinking about chunking.

Solution:
Use a small-to-big or parent-child retrieval pattern.

The idea:

• retrieve using small, precise units,
• optionally expand to a parent chunk when the small unit is relevant,
• include only as much parent context as needed.

Example structure:

Document
  → Section
      → Passage
          → Sentence or small chunk

You embed the small chunks for precise retrieval. When a small chunk is selected, you can decide whether to include its parent passage or section.

A simple policy:

If the small chunk is high-confidence and self-contained:
    use only the small chunk.

If the small chunk contains a pronoun or incomplete rule:
    include the parent passage.

If the answer requires procedure or context:
    include the parent section, but compress it.

This pattern reduces cost because you do not always pay for the largest context unit. You pay for the smallest unit that is sufficient.

Why this works:
Embedding search often works better on focused passages, while LLM reasoning often benefits from surrounding context. Parent-child retrieval separates those needs.

Cost implications

• Smaller retrieval units reduce base token cost.
• Parent expansion increases cost selectively.
• Compression can limit parent expansion cost.
• Metadata can tell the system which parent level to include.

This is especially useful for:

• legal documents,
• technical manuals,
• policies,
• long-form documentation,
• contracts,
• support runbooks.

🧠 The important part:
Chunking is not just “split text into pieces.” It is a cost and quality contract between retrieval precision and context completeness.

8. Metadata filters and query routing shrink the search space

Scenario:
Your assistant answers questions over product docs, billing policies, API references, and customer-specific configuration. Every query searches everything. A billing question retrieves irrelevant API chunks. An API question retrieves marketing copy.

Why it matters:
The more irrelevant material you retrieve, the more top-K and compression have to fight against the problem. Metadata filtering and routing prevent the bad candidates from entering the pool in the first place.

Solution:
Attach useful metadata to chunks and use it during retrieval.

Useful metadata includes:

• tenant ID,
• product name,
• document type,
• access group,
• version,
• locale,
• effective date,
• source system,
• support tier.

Then route the query before retrieving.

A simple router might classify the query into:

billing
api
account
security
general
no_retrieval

If the query is clearly billing-related, search only billing documents. If the query is a greeting or small talk, skip retrieval entirely.

Example routing logic:

def choose_retrieval_scope(query: str, classifier) -> str:
    intent = classifier(query)

    if intent == "billing":
        return "billing_docs"

    if intent == "api":
        return "api_docs"

    if intent == "account":
        return "account_docs"

    if intent == "small_talk":
        return "none"

    return "general_docs"

The classifier can be:

• a rules-based system,
• a small model,
• an LLM with constrained output,
• a keyword router,
• a hybrid of several methods.

Metadata filtering example

Instead of:

search all documents

You do:

search documents where tenant_id = current_tenant
and doc_type = "billing_policy"
and status = "published"
and version = "current"

This improves precision and reduces prompt cost because fewer irrelevant chunks compete for top-K.

Why this works:
Filters reduce the candidate pool. Routing reduces unnecessary retrieval. Both make downstream optimizations easier.

💡 Practical note:
A “no retrieval” path is underrated. If the user asks, “What can you do?” or “Thanks,” you do not need to search a vector database.

9. Evaluate before squeezing: cost cuts can quietly destroy quality

Every cost optimization in RAG has a shadow risk.

• Caching can return stale answers.
• Lower top-K can reduce recall.
• Reranking can add latency.
• Compression can remove critical details.
• Routing can misclassify the query.

That means cost reduction needs an evaluation harness.

Scenario:
You reduce top-K from 8 to 3. Cost per request drops. But users start asking follow-up questions because the first answer is incomplete. Total interactions per resolved issue increases. The system is cheaper per request but worse overall.

Why it matters:
The metric you want is not merely “cost per request.” It is closer to:

cost per acceptable answer

or

cost per resolved user intent

What to measure

Useful metrics include:

• retrieval recall@K,
• retrieval precision@K,
• groundedness,
• citation accuracy,
• answer relevance,
• refusal rate,
• hallucination rate,
• average prompt tokens,
• p50/p95 latency,
• cost per query,
• cost per acceptable answer,
• user follow-up rate.

You do not need perfect evaluation on day one, but you need a repeatable set of test queries.

A minimal evaluation set should include:

• easy factual questions,
• multi-source questions,
• ambiguous questions,
• questions with no answer in the corpus,
• permission-sensitive questions,
• queries with paraphrases,
• queries where the correct answer is “I don’t know,”
• high-risk domain questions.

Shadow mode

Before enabling aggressive optimization for all users, run it in shadow mode.

Production request
  → existing RAG pipeline returns answer
  → optimized pipeline runs in parallel
  → compare retrieval sets, prompt tokens, and answers

This lets you see whether the optimized pipeline would have retrieved the right evidence without exposing users to regressions.

A useful evaluation rule

If a cost optimization changes the answer, inspect whether it changed:

1. from wrong to right,
2. from right to right but cheaper,
3. from right to wrong,
4. from wrong to wrong but cheaper.

Only the first two are improvements.

🔍 Why this matters:
RAG cost optimization is not a single setting. It is a series of trade-offs that must be validated against your actual corpus and user queries.

Choosing the right combination

The best cost-reduction strategy depends on where your cost is coming from.

If ingestion cost is the main problem

Use:

• exact embedding caches,
• content deduplication,
• chunk-level hashing,
• incremental indexing,
• unchanged-document detection.

This is common when documents are reprocessed frequently or contain repeated boilerplate.

If repeated user queries are the main problem

Use:

• exact query caching,
• semantic caching for stable FAQs,
• tenant-scoped cache keys,
• short TTLs for changing data.

This is common in support assistants and documentation bots.

If prompt token cost is the main problem

Use:

• top-K policies,
• token budgets,
• reranking,
• context compression,
• smaller retrieval units,
• parent-child retrieval.

This is the most common production problem.

If retrieval quality is unstable

Do not aggressively cut top-K first. Improve retrieval:

• better metadata,
• hybrid search,
• reranking,
• query rewriting,
• better chunking,
• better embedding model selection.

Reducing K on a weak retriever can make the system cheaper and noticeably worse.

Comparison of techniques

Technique	Primary cost reduced	Complexity	Risk	Best use case
Exact embedding cache	Ingestion/query embeddings	Low	Invalidation mistakes	Re-indexing, repeated chunks
Exact query cache	Embedding and LLM calls	Low to medium	Stale answers	FAQ-style repeated queries
Semantic cache	Repeated intent handling	Medium	Wrong cache hits	Stable, low-risk FAQs
Top-K policy	Prompt tokens	Low to medium	Recall loss	Over-retrieval
Reranking	Prompt tokens, quality	Medium	Latency, rerank cost	Noisy retrieval pools
Extractive compression	Prompt tokens	Medium	Lost evidence	Long chunks
Generative compression	Prompt tokens	Medium to high	Hallucination	Summaries, low-risk answers
Metadata filtering	Retrieval and prompt cost	Medium	Misrouting	Multi-domain or tenant data
Parent-child retrieval	Prompt tokens, quality	High	Complexity	Long documents, policies

What I would choose for a typical production assistant

For a typical internal or support RAG system, I would start with:

1. Exact embedding cache for ingestion.
2. Metadata filtering by tenant and document type.
3. Retrieve 20 to 40 candidates.
4. Rerank to 3 to 5 final chunks.
5. Apply a token budget to final context.
6. Use extractive compression only if chunks are long.
7. Add semantic caching later for stable FAQs, not for account-specific data.
8. Evaluate with a golden query set before and after every change.

That combination usually avoids the worst failure modes: paying for too much context, retrieving irrelevant text, and caching answers that should not be cached.

Production checklist

Before shipping cost optimizations for RAG, verify the following.

Caching

• Embedding cache keys include model and cache version.
• Query cache keys include tenant and access scope where relevant.
• Semantic cache has a high similarity threshold.
• Cache entries have TTLs appropriate to data freshness.
• Cache hits are monitored for quality regressions.
• Cross-tenant cache leakage is impossible by design.

Retrieval

• Top-K is not hardcoded without reason.
• Retrieval selection considers score, token budget, and query type.
• Low-confidence queries have a fallback path.
• Metadata filters reduce the candidate pool.
• Queries that do not need retrieval skip retrieval.

Context size

• Final prompt has a token budget.
• Chunk size is tuned for both retrieval and answer quality.
• Long chunks are compressed or selectively expanded.
• Citations or source references survive compression.
• Compression does not rewrite factual evidence unless necessary.

Quality

• There is a golden query set.
• Retrieval recall is measured before and after cost changes.
• Answer groundedness is checked.
• Cost per acceptable answer is tracked.
• Shadow mode is used for aggressive optimizations.

RAG cost reduction is not about turning one knob. It is about building a system that retrieves less noise, caches safely, and preserves the evidence the model needs to answer correctly. Done well, it makes the system cheaper, faster, and more reliable at the same time.