Your product docs changed. Three pages got rewritten, two got deleted, one new FAQ was added. Your RAG pipeline's answer? Re-embed all 47,000 chunks. That's 40 minutes of compute, $3.20 in API costs, and a 40-minute window where your search quality is inconsistent because half the index is stale.
You didn't build a RAG system to rebuild it every time someone edits a typo.
The dirty secret of most RAG tutorials is that they treat the knowledge base as immutable. You ingest, you embed, you query. Done. But production knowledge bases are alive. Documents get updated, deprecated, restructured. And the naive "re-embed everything" approach doesn't survive contact with a content team that ships updates daily.
This article covers the patterns I've seen work in production for keeping a RAG index current without paying the full embedding cost on every change. Some are simple. Some require rethinking your chunking strategy. All of them assume you'd rather spend your compute budget on actual inference, not redundant embedding calls.
📋 Table of Contents
- The Real Cost of "Just Re-Embed Everything"
- 1. Content Hashing: The Change Detection Gatekeeper
- 2. Chunk-Level Upsert With Stable Identifiers
- 3. The Tombstone Pattern for Deleted Content
- 4. Metadata Versioning Instead of Deletion
- 5. Change Data Capture Pipelines for High-Volume Sources
- 6. Handling the Dreaded Embedding Model Upgrade
- 7. Hybrid Keyword + Vector Indexes as an Escape Hatch
- 8. Shadow Index Swaps for Zero-Downtime Rebuilds
- Choosing Your Strategy: A Decision Matrix
The Real Cost of "Just Re-Embed Everything"
Before jumping into solutions, let's make the problem concrete.
Scenario: You maintain an internal knowledge base with 12,000 documents. Your chunking strategy produces roughly 8 chunks per document. That's ~96,000 chunks. You use OpenAI's text-embedding-3-small at $0.02 per 1M tokens, and your average chunk is 400 tokens.
A full re-embed costs about $0.77 and takes 8–12 minutes depending on rate limits and concurrency. Fine for a weekly rebuild. Not fine when:
- Your CMS webhook fires 30 times a day
- Your legal team updates compliance docs and needs changes reflected within the hour
- You're embedding with a local model and the GPU is shared with inference
The problem isn't just cost. It's consistency window. During a full re-embed, queries hit a partially-updated index. You get answers that reference the old version of paragraph three and the new version of paragraph seven. For most use cases, that's worse than a slightly stale but internally consistent index.
The goal, then: update only what changed, keep everything else untouched, and never leave the index in a half-migrated state.
1. Content Hashing: The Change Detection Gatekeeper
Scenario: Your ingestion pipeline pulls documents from a Confluence space, a Git repo, and an S3 bucket. Every sync cycle, you re-download everything and re-embed everything because you don't actually know what changed.
Why it matters: Without change detection, every optimization downstream is irrelevant. You need a fast, cheap way to answer: "Did this document actually change since the last time I embedded it?"
Solution: Compute a content hash (SHA-256 works fine) for each document at ingestion time. Store it alongside the document's metadata. On the next sync, compare hashes before doing anything expensive.
import hashlib
from dataclasses import dataclass
@dataclass
class DocumentState:
doc_id: str
content_hash: str
chunk_count: int
last_embedded_at: str
def compute_hash(content: str) -> str:
return hashlib.sha256(content.encode("utf-8")).hexdigest()
def detect_changes(documents: list[dict], existing_states: dict[str, DocumentState]) -> tuple[list, list, list]:
"""Returns (new_docs, changed_docs, deleted_doc_ids)"""
new, changed = [], []
seen_ids = set()
for doc in documents:
doc_id = doc["id"]
seen_ids.add(doc_id)
content_hash = compute_hash(doc["content"])
if doc_id not in existing_states:
new.append(doc)
elif existing_states[doc_id].content_hash != content_hash:
changed.append(doc)
# else: unchanged, skip entirely
deleted = [
doc_id for doc_id in existing_states
if doc_id not in seen_ids
]
return new, changed, deleted
Why this works: SHA-256 on a 10KB document takes microseconds. You can hash 100,000 documents in under a second. Compare that to even a single embedding API call. The hash acts as a cheap gatekeeper—only documents that fail the hash check proceed to chunking and embedding.
💡 Practical note: Hash the content, not the file's modification timestamp. Timestamps lie. A CI pipeline might rewrite a file with identical content and update the mtime. A user might edit a file and revert it. Content hashing is the only check that tells you whether the embeddings would actually differ.
One subtlety: if your chunking strategy depends on document structure (headings, paragraph boundaries), hash the raw content before chunking. Two documents with identical text but different heading structures might chunk differently, but the hash catches the text change and you re-chunk from there.
2. Chunk-Level Upsert With Stable Identifiers
Scenario: A single paragraph in a 40-page PDF changed. Your pipeline detects the document changed (good), but now it re-embeds all 320 chunks from that PDF because your vector DB treats the document as an atomic unit.
Why it matters: Document-level granularity wastes 95% of your embedding budget on unchanged content. You need chunk-level addressing.
Solution: Assign each chunk a deterministic, stable identifier derived from its parent document and its position or content. Then use your vector database's upsert operation to replace only the chunks that changed.
def generate_chunk_id(doc_id: str, chunk_index: int, chunk_content: str) -> str:
"""Stable ID: same content at same position = same ID."""
content_sig = hashlib.md5(chunk_content.encode()).hexdigest()[:8]
return f"{doc_id}::chunk_{chunk_index}::{content_sig}"
def upsert_changed_chunks(vector_store, doc_id: str, new_chunks: list[str], old_chunk_ids: list[str]):
# Generate new chunk IDs
new_ids = [generate_chunk_id(doc_id, i, c) for i, c in enumerate(new_chunks)]
# Find chunks to remove (old IDs not in new set)
ids_to_delete = [cid for cid in old_chunk_ids if cid not in set(new_ids)]
# Upsert only new/changed chunks
vectors = embed_batch(new_chunks) # Only embeds what's actually new
vector_store.upsert(
ids=new_ids,
vectors=vectors,
metadatas=[{"doc_id": doc_id, "chunk_index": i} for i in range(len(new_chunks))]
)
# Remove stale chunks
if ids_to_delete:
vector_store.delete(ids=ids_to_delete)
This works with Pinecone's upsert, Qdrant's upsert, Weaviate's object update, or pgvector with an ON CONFLICT clause. The key is that your chunk ID is deterministic. If the same text appears at the same position, you get the same ID, and the upsert is a no-op at the storage level.
The tricky part: What happens when an edit shifts all subsequent chunks? If someone inserts a paragraph at the top, every chunk index shifts. Your chunk_17 now contains what used to be chunk_16. With content-signature-based IDs, this resolves itself—the old chunk_17 gets deleted (its content hash no longer matches), and the new content at position 17 gets a fresh ID. You pay for re-embedding the shifted chunks, but only those, not the entire document.
⚠️ Gotcha: If you use purely positional IDs (
doc123::chunk_5) without content signatures, a single insertion re-embeds every subsequent chunk because their IDs now map to different content. Always include a content fingerprint in the chunk ID.
3. The Tombstone Pattern for Deleted Content
Scenario: A product feature was deprecated. The documentation page got deleted from your CMS. But your vector store still has 14 chunks from that page, and your RAG system keeps recommending the deprecated approach to users.
Why it matters: Deleting from the source doesn't delete from the vector index. If your sync pipeline only processes "documents that exist," deleted content becomes a ghost.
Solution: Treat deletions as first-class events. Rather than immediately purging vectors, mark them as inactive with a tombstone metadata flag. This gives you an audit trail and a rollback window.
def handle_deleted_documents(vector_store, deleted_doc_ids: list[str]):
"""Soft-delete: mark as inactive rather than purging."""
for doc_id in deleted_doc_ids:
vector_store.update_metadata(
filter={"doc_id": doc_id},
metadata_update={"status": "tombstoned", "deleted_at": utc_now_iso()}
)
def query_active_only(vector_store, query_vector: list[float], top_k: int = 5):
"""Always filter out tombstoned chunks at query time."""
return vector_store.query(
vector=query_vector,
top_k=top_k,
filter={"status": {"$ne": "tombstoned"}}
)
Then run a periodic hard-delete job (nightly, weekly) that purges chunks tombstoned more than 7 days ago. This gives you a safety window if a deletion was accidental.
Why not just hard-delete immediately? Two reasons. First, race conditions: a query might be in-flight when you delete, and depending on your vector DB's consistency model, it might return the deleted chunk anyway. Second, operational safety: "Oops, that page wasn't supposed to be deleted" is a conversation you will have. The tombstone pattern makes undoing that a metadata flip instead of a re-embedding job.
4. Metadata Versioning Instead of Deletion
Scenario: Your legal team updates the terms of service. The old version needs to stay queryable for customers who agreed to it last month. The new version needs to be the default answer going forward. But they're the same document ID in your system.
Why it matters: This is the "living document" problem. You can't just overwrite the old embedding because historical queries need it. You can't keep both active because new queries would get contradictory answers.
Solution: Embed each version as a separate set of chunks with version metadata. Use a valid_from / valid_until temporal filter at query time.
def embed_versioned_document(vector_store, doc_id: str, content: str, version: int):
chunks = chunk_document(content)
vectors = embed_batch(chunks)
vector_store.upsert(
ids=[f"{doc_id}::v{version}::chunk_{i}" for i in range(len(chunks))],
vectors=vectors,
metadatas=[{
"doc_id": doc_id,
"version": version,
"valid_from": utc_now_iso(),
"valid_until": None, # current version
"status": "active"
} for i in range(len(chunks))]
)
# Close out previous version
vector_store.update_metadata(
filter={"doc_id": doc_id, "version": {"$lt": version}, "valid_until": None},
metadata_update={"valid_until": utc_now_iso(), "status": "superseded"}
)
def query_current_version(vector_store, query_vector, doc_id: str = None):
filters = {"status": "active"}
if doc_id:
filters["doc_id"] = doc_id
return vector_store.query(vector=query_vector, top_k=5, filter=filters)
This pattern costs more storage (you're keeping old vectors around), but it eliminates re-embedding entirely for version updates. The old chunks stay exactly as they are. You just stop querying them.
🔍 Why this matters: This is also your answer to "the CEO wants to know what the knowledge base said about refunds six months ago." With versioned metadata, that's a filter change, not an archaeology project.
The trade-off is index size. If your documents change frequently and you keep all versions, your vector store grows linearly with edit history. For most knowledge bases, this is fine—storage is cheap relative to embedding compute. But set a retention policy. Prune versions older than your compliance window.
5. Change Data Capture Pipelines for High-Volume Sources
Scenario: Your knowledge base is sourced from a PostgreSQL database with 200,000 rows, each representing a support ticket resolution. New rows arrive at ~50/minute. You can't poll the whole table every five minutes and hash everything.
Why it matters: Polling doesn't scale. At high ingestion rates, you need event-driven updates that process only the rows that actually changed.
Solution: Use Change Data Capture (CDC) to stream inserts, updates, and deletes from your source database directly into your embedding pipeline.
# Using Debezium-style CDC events (e.g., from Kafka Connect)
def process_cdc_event(event: dict, vector_store, embedding_client):
op = event["op"] # 'c' = create, 'u' = update, 'd' = delete
row = event["after"] if op != "d" else event["before"]
doc_id = f"ticket_{row['id']}"
if op == "d":
vector_store.delete(filter={"doc_id": doc_id})
return
content = f"Q: {row['question']}\nA: {row['resolution']}"
content_hash = compute_hash(content)
# Check if this exact content is already embedded
existing = vector_store.fetch_metadata(filter={"doc_id": doc_id})
if existing and existing.get("content_hash") == content_hash:
return # No actual change, skip
chunks = chunk_document(content)
vectors = embedding_client.embed(chunks)
vector_store.upsert(
ids=[f"{doc_id}::chunk_{i}" for i in range(len(chunks))],
vectors=vectors,
metadatas=[{"doc_id": doc_id, "content_hash": content_hash, "updated_at": event["ts_ms"]}]
)
The architecture looks like:
Source DB → Debezium / pg_logical → Kafka → Embedding Worker → Vector Store
(batched, 50ms window)
Why this works: You never touch unchanged data. The CDC stream is your change detection. Each event is already scoped to a single row. Your embedding worker batches events in small windows (50–200ms) to amortize API call overhead without introducing meaningful latency.
Production considerations:
- Ordering: CDC events for the same row must be processed in order. Partition your Kafka topic by document ID.
- Idempotency: Workers will restart. Use the content hash check above so replayed events don't cause duplicate embeddings.
- Backpressure: If the embedding API rate-limits you, buffer in Kafka. Don't drop events. The consumer group offset tracks your position.
🚨 Production warning: If your CDC source can emit out-of-order events (network partitions, multi-region setups), include a monotonic sequence number or timestamp in the row, and skip events older than what you've already processed. A stale update overwriting a fresh one is a subtle, maddening bug.
6. Handling the Dreaded Embedding Model Upgrade
Scenario: You've been running text-embedding-3-small for a year. OpenAI releases a new model with 30% better retrieval performance. You want to switch. But you have 2 million chunks. Re-embedding everything takes 6 hours and costs $150. You can't take the system offline that long.
Why it matters: This is the one scenario where you genuinely do need to re-embed everything—old vectors from model A are incompatible with query vectors from model B. You can't mix them. But you can do it without downtime.
Solution: Blue-green embedding migration. Build the new index in parallel, then cut over atomically.
# Phase 1: Dual-write (new embeddings go to both indexes)
def dual_write(chunk_id: str, content: str):
old_vector = embed_with_model("text-embedding-3-small", content)
new_vector = embed_with_model("text-embedding-3-large", content)
old_index.upsert(ids=[chunk_id], vectors=[old_vector])
new_index.upsert(ids=[chunk_id], vectors=[new_vector])
# Phase 2: Backfill (process existing chunks in batches)
def backfill_new_index(old_index, new_index, batch_size=500):
cursor = old_index.list_ids()
for batch in iterate_batches(cursor, batch_size):
chunks = old_index.fetch_content(batch)
vectors = embed_with_model_batch("text-embedding-3-large", chunks)
new_index.upsert(ids=batch, vectors=vectors)
# Checkpoint progress for resumability
save_checkpoint(last_id=batch[-1])
# Phase 3: Cutover (flip query routing)
def switch_query_target(config):
config["active_index"] = "new_index"
# Old index stays readable for 48h rollback window
The backfill runs at whatever pace your budget allows. With rate-limited API access, it might take a day. That's fine—during that time, queries still hit the old index. New writes go to both. Once backfill completes, you validate retrieval quality on a held-out test set, then flip the query router.
The critical insight: you never re-embed "every document" in one shot. You backfill incrementally, with checkpointing, over hours or days, while the system stays live.
💡 Practical note: Before committing to a full migration, embed a representative sample (say, 5,000 chunks) with the new model and run your evaluation suite. If retrieval recall@5 doesn't improve meaningfully on your data, skip the migration. Vendor benchmarks are not your benchmarks.
7. Hybrid Keyword + Vector Indexes as an Escape Hatch
Scenario: Your RAG system uses a hybrid approach—BM25 keyword search combined with vector similarity. A document's metadata changes (tags, category, author) but the body text doesn't. Your pipeline re-embeds the chunk because the "document changed." The vector is identical. You wasted an API call.
Why it matters: In hybrid systems, not all changes require re-embedding. Metadata changes affect filtering and keyword scoring but not the vector itself.
Solution: Separate your update paths. Vector re-embedding only triggers on content changes. Metadata updates propagate through a cheaper path.
def handle_document_update(doc_id: str, old_doc: dict, new_doc: dict, vector_store):
content_changed = compute_hash(old_doc["body"]) != compute_hash(new_doc["body"])
metadata_changed = old_doc["metadata"] != new_doc["metadata"]
if content_changed:
# Full re-chunk and re-embed pipeline
re_embed_document(vector_store, doc_id, new_doc["body"])
if metadata_changed and not content_changed:
# Just update metadata in the vector store—no embedding call
vector_store.update_metadata(
filter={"doc_id": doc_id},
metadata_update=new_doc["metadata"]
)
if metadata_changed:
# Update the keyword/BM25 index separately
keyword_index.update(doc_id, new_doc["metadata"], new_doc["body"])
This separation becomes more important as your system matures. You might have:
- Vector index (content embeddings)
- Keyword index (BM25 / full-text search)
- Metadata store (filters, access control, tags)
- Graph index (entity relationships)
Each has different update triggers. A tag change hits metadata and keyword indexes. A paragraph edit hits vector and keyword. A permissions change hits only metadata. Treating all changes as "re-embed everything" conflates these concerns.
8. Shadow Index Swaps for Zero-Downtime Rebuilds
Scenario: Despite all your incremental updates, the index has accumulated fragmentation. Chunk IDs from three different schema versions. Tombstoned entries taking up space. Metadata inconsistencies from a buggy migration six months ago. You need a clean rebuild, but you can't take search offline.
Why it matters: Sometimes incremental patches accumulate enough technical debt that a clean rebuild is genuinely the right call. The question is how to do it without a maintenance window.
Solution: Build a completely fresh index in the background (the "shadow"), validate it against production queries, then swap the alias.
# Using an alias/pointer pattern (supported by Elasticsearch, Qdrant, Weaviate, etc.)
def create_shadow_index(source_documents):
shadow_name = f"knowledge_base_shadow_{int(time.time())}"
create_empty_index(shadow_name, schema=CURRENT_SCHEMA)
for batch in iterate_documents(source_documents, batch_size=1000):
chunks = [chunk_document(doc["content"]) for doc in batch]
flat_chunks = [c for doc_chunks in chunks for c in doc_chunks]
vectors = embed_batch(flat_chunks)
shadow_index.upsert(...)
return shadow_name
def validate_shadow(shadow_name, test_queries: list[str], min_recall: float = 0.95):
"""Run held-out queries against shadow and compare to production."""
for query in test_queries:
prod_results = query_index("knowledge_base_prod", query)
shadow_results = query_index(shadow_name, query)
recall = compute_recall(prod_results, shadow_results)
if recall < min_recall:
raise ValidationFailed(f"Shadow recall {recall} below threshold")
def swap_alias(alias: str, old_index: str, new_index: str):
"""Atomic alias swap. Queries never see a gap."""
vector_store.update_alias(
alias=alias,
remove_index=old_index,
add_index=new_index
)
The alias pattern means your application always queries knowledge_base_prod (an alias), never a concrete index name. The swap is a single atomic operation at the vector DB level. Queries in-flight against the old index complete normally. New queries hit the shadow.
When to use this vs. incremental updates:
| Signal | Incremental update | Shadow rebuild |
|---|---|---|
| < 5% of documents changed | ✅ | Overkill |
| Schema/chunking strategy changed | ❌ Can't patch | ✅ |
| Index fragmentation > 20% | Getting slow | ✅ |
| Embedding model changed | ❌ Incompatible vectors | ✅ |
| Need to fix a bad migration | Painful | ✅ |
🧠 The important part: The shadow build still benefits from your embedding cache. If you've stored
(content_hash → vector)pairs in a cache layer (Redis, S3, a local SQLite file), the shadow build can skip API calls for content that hasn't changed. You're rebuilding the index structure, not recomputing every vector. This can cut a "full rebuild" from 6 hours to 45 minutes.
Choosing Your Strategy: A Decision Matrix
There's no single answer. The right approach depends on your update frequency, data volume, and consistency requirements. Here's how I think about it:
Update frequency:
- A few docs per week: Content hashing + chunk-level upsert (sections 1–2) is all you need. Keep it simple.
- Dozens of docs per day: Add tombstoning (section 3) and metadata versioning (section 4) to handle deletions and living documents gracefully.
- Hundreds of events per minute: You need CDC (section 5). Polling won't keep up.
Change type:
- Content edits: Incremental re-embedding of affected chunks only.
- Metadata-only changes: Update metadata in-place. Don't touch vectors.
- Deletions: Tombstone first, hard-delete later.
- Model upgrades: Shadow index with backfill (section 6). No way around re-embedding, but no downtime.
- Schema changes / accumulated debt: Full shadow rebuild (section 8).
Consistency requirements:
- Eventual consistency is fine (hours): Batch your updates. Run a sync job every 30 minutes. Simple, cheap, effective.
- Near-real-time (seconds): CDC with streaming embedding workers. More infrastructure, tighter operational burden.
- Transactional (the doc is updated, the index must reflect it before the user's next query): You're in hard territory. Consider synchronous embedding in the write path, but understand the latency cost. For most RAG applications, 30–60 seconds of staleness is invisible to users.
The question I'd ask before building any of this: What's the actual user-facing impact of a stale answer? If your knowledge base is product documentation and a feature description is 2 hours out of date, nobody notices. If it's a medical decision support system, 2 hours might be unacceptable. Let that answer drive your architecture, not the other way around.
One Last Thing: Cache Your Embeddings
Regardless of which strategy you adopt, maintain an embedding cache keyed by (content_hash, model_name, model_version).
def embed_with_cache(content: str, model: str) -> list[float]:
cache_key = f"{compute_hash(content)}:{model}"
cached = cache_store.get(cache_key)
if cached:
return cached
vector = embedding_client.embed(content, model=model)
cache_store.set(cache_key, vector, ttl=90_days)
return vector
This protects you in every scenario:
- Re-running a failed pipeline? Cache hits for unchanged chunks.
- Shadow rebuild? Most vectors are already computed.
- Rollback to a previous embedding model? Those vectors are still in cache.
- Accidentally deleted a chunk and need to restore it? One cache lookup, not one API call.
The cache is the cheapest insurance policy in your entire RAG infrastructure. A 50GB Redis instance or an S3 bucket with a SQLite index will hold millions of cached vectors. The ROI is absurd.
The knowledge base is never "done." It's a living system that needs a living update strategy. The patterns above aren't mutually exclusive—most production systems I've seen combine three or four of them. Start with content hashing and chunk-level upsert. Add complexity only when your update volume or consistency requirements demand it. And whatever you do, stop re-embedding 96,000 chunks because someone fixed a comma on page 47.