A user types: "Can we turn that off for a single workspace?"
Your retriever embeds that sentence, finds the nearest vectors, and hands the generator a confident pile of documentation about workspace creation. The answer is fluent, wrong, and cited.
So you add multi-query retrieval. The model produces three polished variations:
- Can a workspace disable this feature?
- Is per-workspace opt-out supported?
- How do I turn this off for one workspace?
All three still contain an unresolved this. You've paid for three embeddings of the same ambiguity.
Now try a rewrite with the previous four turns in context: "How do I disable SAML enforcement for one workspace on the Enterprise plan?" The retriever finds the right page on the first try.
But the reverse failure is just as common. A rewrite confidently expands "the old API" into a specific deprecated version the user never mentioned, and you've retrieved the wrong answer with higher precision than before.
Here's the distinction worth keeping in your head: query rewriting changes what you're asking. Multi-query retrieval changes how many ways you ask it. They don't compete for the same job, and treating them as alternatives is how teams end up paying triple for a retrieval step that still misses.
TL;DR
- Rewrite when the query is ambiguous, context-dependent, or missing scope. Fan out when the query is clear but the corpus speaks a different vocabulary.
- Multi-query retrieval without a rerank stage mostly buys you a bigger, noisier context window. The recall gain only converts into answer quality when something scores the merged candidates against the actual intent.
- Three angles beat ten paraphrases. Paraphrases from one model share the same blind spots.
- Evaluate every variant at the same final context budget. Comparing a 1-chunk baseline against a 60-chunk fan-out measures nothing useful.
📋 Table of Contents
- 1. Diagnose the Failure Before You Add an LLM to the Query Path
- 2. Contextual Rewriting Turns That Thing Into a Searchable Sentence
- 3. The Rewrite That Quietly Changes the Question
- 4. Multi-Query Retrieval Needs Angles, Not Paraphrases
- 5. Reciprocal Rank Fusion Is a Budget Tool, Not a Scoring Model
- 6. The Reranker Is Where Multi-Query Actually Pays Off
- 7. When More Queries Make Your Answers Worse
- 8. HyDE, Decomposition, and Agentic Search Are Not the Same Thing
- 9. Evaluate at a Fixed Context Budget or You Are Fooling Yourself
- 10. A Routing Policy You Can Ship on a Friday
1. Diagnose the Failure Before You Add an LLM to the Query Path
Scenario: Two engineers build two demos. One adds contextual rewriting and fixes every "and in the EU?" follow-up. Another adds multi-query retrieval and fixes every "my pod keeps crashing" question that the docs describe as OOMKilled. Each demo convinces a different stakeholder. Neither fixes the other's failure case.
That's not a tooling disagreement. It's two different failure modes wearing the same symptom: the answer was wrong.
| Failure mode | What the query looks like | What actually fixes it |
|---|---|---|
| Unresolved reference or ellipsis | "and for the other region?" | Contextual rewriting |
| Vocabulary mismatch | "crash" vs. docs saying OOMKilled | Multi-query, synonym/hybrid search |
| Compound or multi-hop intent | "Which plan supports the feature we use for SSO?" | Decomposition, not paraphrase |
| Hard constraint | "v2.3, not v3, excluding EU tenants" | Structured filters + literal preservation |
| Nothing in the corpus answers it | A question about a feature you don't have | Abstention, not expansion |
The diagnostic test I'd use on a query log is embarrassingly simple. Read the query alone, as a competent human who hasn't seen the conversation:
- If you'd ask "what do you mean?", you have a rewriting problem.
- If you'd say "I understand, but I'd search three different ways", you have a fan-out problem.
- If you'd say "I need to look something up first, then search", you have a decomposition problem.
Everything below follows from that split.
Before writing any transformation code, pull 100 real queries — including multi-turn follow-ups, including ones your corpus can't answer — and label the chunks that should have been retrieved. Don't generate your whole eval set from your own documents. That selects for the vocabulary your corpus already uses, which is precisely the case where query transformation looks most impressive and helps least.
🔍 Why this matters: If most of your failures are unresolved references or dropped constraints, a larger fan-out budget is being spent on the wrong problem. More vectors cannot disambiguate a pronoun.
2. Contextual Rewriting Turns That Thing Into a Searchable Sentence
The job of a contextual rewriter is narrow: turn a conversational turn into a standalone search query using the conversation as a dictionary, not as a source of new facts.
It runs after authentication, before embedding. It does not answer anything.
Scenario: Your support bot has been discussing SAML configuration for six turns. The user says "can we skip that for one workspace?" Embedding that sentence directly is close to meaningless — the informative tokens (that, one, workspace) are either generic or already saturated with unrelated matches.
Solution: Send the last few turns plus the current turn to a small, fast model with a restrictive instruction and a structured output schema.
import json
import os
from openai import OpenAI
client = OpenAI()
REWRITE_MODEL = os.environ["REWRITE_MODEL"] # a small model with structured outputs
SYSTEM = """Rewrite the user turn into a standalone search query.
- Use the conversation only to resolve references and scope.
- Copy product names, error codes, versions and dates verbatim.
- Preserve negation, exclusions, and modality (can / must / might).
- Never answer the question, add facts, or guess an ambiguous referent.
- If the turn cannot be resolved, set resolvable to false and return the turn."""
SCHEMA = {
"name": "rewritten_query",
"strict": True,
"schema": {
"type": "object",
"additionalProperties": False,
"properties": {
"query": {"type": "string"},
"resolvable": {"type": "boolean"},
"resolved_references": {"type": "array", "items": {"type": "string"}},
},
"required": ["query", "resolvable", "resolved_references"],
},
}
def rewrite(turn: str, history: list[dict], as_of: str) -> dict:
transcript = "\n".join(f"{m['role']}: {m['content']}" for m in history[-6:])
resp = client.chat.completions.create(
model=REWRITE_MODEL,
temperature=0,
response_format={"type": "json_schema", "json_schema": SCHEMA},
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"Date: {as_of}\n\n{transcript}\n\nTurn: {turn}"},
],
)
return json.loads(resp.choices[0].message.content)
For the SAML example, a good result looks like:
{
"query": "How do I disable SAML enforcement for a single workspace on the Enterprise plan?",
"resolvable": true,
"resolved_references": ["that -> SAML enforcement", "the plan -> Enterprise"]
}
Three implementation details carry most of the weight.
The schema is not decoration. A free-text rewrite gives you no way to distinguish "the model resolved the reference" from "the model invented one." An explicit resolvable flag gives you a routing target: false means clarify or fall back, not retrieve confidently. And resolved_references is a cheap audit trail — you can check that each resolved entity actually appears in the supplied history.
Six turns, not the whole transcript. Old topics pollute the embedding. A conversation that started with billing and moved to SSO will happily retrieve billing documents if you paste all forty turns into the rewrite prompt. For long sessions, carry a rolling summary plus the recent turns; never embed the transcript itself as the query.
The date comes from your server. "Last quarter" is not resolvable from a model's training data. Inject as_of explicitly, or instruct the rewriter to leave relative time expressions untouched and handle them as metadata filters. Letting the model guess the calendar is a quiet, recurring source of wrong answers.
⚠️ Gotcha:
temperature=0reduces variance; it does not guarantee determinism. If you need reproducible evals, pin the model version and cache rewrites by(history fingerprint, turn, prompt version, model version).
3. The Rewrite That Quietly Changes the Question
Rewriting's failure mode is worse than failing to rewrite. A failed rewrite returns a mediocre query. A successful-but-wrong rewrite returns a precise wrong query, and every downstream component obligingly optimizes for it.
The patterns repeat:
| Original | Rewrite | Damage |
|---|---|---|
| "Can we export invoices without attaching VAT?" | "How to export invoices with VAT" | Negation inverted |
| "Does v2.3 support webhook retries?" | "How to configure retries in v3" | Version substituted |
| "Where do we store EU customer data?" | "Where is customer data stored?" | The constraint was the question |
| "What are the limits?" after a rate-limit discussion | "What are the attachment size limits?" | Plausible, unverified scope guess |
Notice that the last row has no lexical error at all. The model made a reasonable bet about an ambiguous referent. In a retrieval system, reasonable bets need a fallback.
Solution: guardrails ordered from cheapest to most expensive.
First, preserve literals. Version strings, error codes, CVE identifiers, dates, and quoted names should survive verbatim. A tokenizer-and-LLM round trip is not a safe way to copy ERR-503.
import re
LITERAL = re.compile(r"\bv?\d+\.\d+(?:\.\d+)?\b|\b[A-Z]+-\d+\b|\b\d{4}-\d{2}-\d{2}\b")
NEGATION = re.compile(r"\b(no|not|never|without|except|excluding|unless|cannot|can't|don't|won't)\b", re.I)
def rewrite_is_safe(original: str, rewritten: str) -> bool:
if not rewritten.strip():
return False
if any(token not in rewritten for token in LITERAL.findall(original)):
return False
if NEGATION.search(original) and not NEGATION.search(rewritten):
return False
return True
Then fail open, never closed:
result = rewrite(turn, history, as_of="2026-02-14")
if not result["resolvable"] or not rewrite_is_safe(turn, result["query"]):
search_queries = [turn] # plus a clarification if ambiguity is material
else:
search_queries = [result["query"]]
This checker is a tripwire, not a semantic proof. "Without VAT" legitimately becomes "VAT excluded," and the naive negation check will flag it. That's fine: review the trips, normalize your cue list, and don't silently delete the check because it occasionally cries wolf. For high-stakes corpora, add a second-stage constraint-preservation check, but understand you're buying an additional model call and its own failure modes.
Second, and non-negotiable: tenant, project, and permission filters come from the authenticated request, never from the rewrite. A rewriter that helpfully expands "our contracts" into "all contracts" has just proposed a security boundary change. Apply ACL filters server-side on every retrieval arm, including speculative ones.
🧠 The important part: When the rewrite is low-confidence, keep the original query as one retrieval arm. It's an inexpensive hedge against your own guardrail being wrong.
4. Multi-Query Retrieval Needs Angles, Not Paraphrases
Scenario: The query is perfectly clear: "Why do webhook deliveries stop after a deploy?" Your multi-query step returns:
- Why do webhook deliveries stop following a deployment?
- What causes webhooks to fail after deploying?
- Reasons webhook delivery halts post-deployment.
Retrieval returns essentially the same twenty chunks three times. You've built a more expensive single query.
The problem is the instruction. "Give me three different ways to say this" asks for stylistic variance. Embedding space doesn't care much about style. It cares about vocabulary, entities, and what a relevant document is about.
Solution: Ask for complementary retrieval intents, each labeled with the axis it varies.
Useful axes in a real product corpus:
- Vocabulary / alias:
webhook delivery→event dispatch→callback - Artifact type: runbook vs. API reference vs. changelog entry
- Causal facet: timeout vs. retry policy vs. egress allowlist
- Representation: natural question vs. keyword-style string
- Language or region, when the corpus is genuinely multilingual
def expand_query(query: str, limit: int = 3) -> list[dict]:
"""One generation call returns a small set of complementary retrieval intents."""
resp = client.chat.completions.create(
model=os.environ["EXPANSION_MODEL"],
temperature=0.4,
response_format={"type": "json_schema", "json_schema": EXPANSION_SCHEMA},
messages=[
{"role": "system", "content": (
"Return up to 3 search queries for this intent. They must differ in "
"vocabulary, artifact type, or causal facet — not in phrasing. "
"Do not add facts, entities, or constraints absent from the intent. "
"Label each with the axis it varies."
)},
{"role": "user", "content": query},
],
)
items = json.loads(resp.choices[0].message.content)["queries"]
seen, out = set(), []
for item in items:
key = " ".join(item["text"].casefold().split())
if key not in seen:
seen.add(key)
out.append(item)
return out[:limit]
EXPANSION_SCHEMA is the same strict JSON pattern as before: an array of {text, angle} objects, all keys required, additionalProperties: false. The angle field earns its keep in debugging — when a variant retrieves garbage, you want to know whether it was supposed to be a vocabulary probe or a facet probe.
Note the temperature choice. Rewriting wanted 0 because ambiguity resolution should be conservative. Expansion tolerates a little sampling because you're deliberately seeking diversity — but this is a knob to test on your eval set, not a law.
Now the part most tutorials get wrong: fan-out does not mean N sequential LLM calls, and it does not mean N embedding round trips.
texts = [q["text"] for q in expand_query(canonical_query)]
embeddings = client.embeddings.create(
model=os.environ["EMBED_MODEL"], input=texts
).data
One batched embedding call. Then run the vector searches concurrently against your existing retriever, with the same server-side ACL filter on every arm. With Qdrant's Python client, that's the query_points() API rather than the older search() helper:
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
qdrant = QdrantClient(url=os.environ["QDRANT_URL"])
def search(vector: list[float], tenant: str, top_k: int = 20):
return qdrant.query_points(
collection_name="docs",
query=vector,
limit=top_k,
query_filter=Filter(must=[
FieldCondition(key="tenant_id", match=MatchValue(value=tenant))
]),
with_payload=True,
).points
The latency shape matters more than the operation count. Generation is serial. Embedding is batched. ANN searches parallelize. So a three-way fan-out is roughly one extra generation hop plus three cheap searches, not three times the end-to-end latency — assuming your implementation actually batches and parallelizes instead of looping.
5. Reciprocal Rank Fusion Is a Budget Tool, Not a Scoring Model
You now have three ranked lists. How do you turn them into one candidate pool?
The tempting answer — average the similarity scores — is quietly broken. Cosine similarity, BM25, and whatever normalized score your hosted retrieval API returns don't share a scale. Averaging them is arithmetic on incomparable units.
Reciprocal Rank Fusion throws the scores away and keeps the ranks:
RRF(d) = sum over queries q of: 1 / (c + rank_q(d))
Rank starts at 1. A document absent from a list contributes nothing to that list. c = 60 is the conventional default from the original RRF work, not a universal optimum.
from collections import defaultdict
def rrf(rankings: list[list[str]], c: int = 60,
weights: list[float] | None = None) -> list[tuple[str, float]]:
scores = defaultdict(float)
for i, ranking in enumerate(rankings):
w = weights[i] if weights else 1.0
for rank, chunk_id in enumerate(ranking, start=1):
scores[chunk_id] += w / (c + rank)
return sorted(scores.items(), key=lambda item: item[1], reverse=True)
Feed it lists of stable chunk IDs, not text. Deduplication happens implicitly: the same chunk appearing in three lists accumulates three contributions. That's the intended behavior — independent confirmation from different angles should raise a candidate.
Three consequences worth internalizing.
RRF is deliberately rank-based, which means it discards confidence too. A query that found an obvious perfect match contributes the same first-place weight as a query that was grasping at straws. If one arm is more trustworthy — usually the original or conservatively rewritten query — express that with weights, or let the reranker sort it out downstream.
RRF scores are not comparable across different numbers of lists. Five queries produce more total score mass than three. So a hardcoded if rrf_score > 0.02: keep threshold silently changes meaning the day someone tunes your fan-out factor. This is a real, annoying class of bug. Use RRF for ordering only; use calibrated reranker scores or an explicit relevance gate for abstention.
The constant c shapes how winner-take-all you are. Small c heavily rewards first places; large c flattens the contribution of deep ranks. Tune it on your eval set alongside top_k per arm, not by intuition.
💡 Practical note: If your vector database already fuses lexical and dense results internally, you now have two fusion layers. That's not automatically wrong, but you should be able to explain what each one is doing. And don't transfer a relevance threshold across them.
6. The Reranker Is Where Multi-Query Actually Pays Off
Here's the claim I'd defend most strongly in this article: multi-query retrieval without a rerank stage is mostly a precision loss.
Work through the arithmetic. Three arms × 20 results = 60 candidates, maybe 40 after deduplicating by chunk ID. Your generator has room for six chunks. What selects those six?
If you take the first six of the RRF list, you've selected by vote count across phrasings. That correlates with relevance, sure — but it also correlates with "which text sits near the center of your embedding model's biases." You expanded coverage and then threw the coverage away at the last step.
The two-stage pattern fixes this: retrieve broadly, then score narrowly.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def select_context(intent: str, candidates: dict[str, str], budget: int = 6):
items = list(candidates.items())
scores = reranker.predict([(intent, text) for _, text in items])
order = sorted(range(len(items)), key=lambda i: scores[i], reverse=True)
return [items[i][0] for i in order[:budget]]
A cross-encoder encodes the query and passage jointly, rather than embedding each independently. That's the mechanical reason it catches things a bi-encoder misses: the difference between "supports retry" and "does not support retry," a version number that doesn't match, a passage about the right feature in the wrong deployment model.
The non-obvious implementation choice is which query you rerank against. Use the canonical rewritten intent, not each variant. Otherwise you've reintroduced at rerank time exactly the phrasing variance you were trying to hedge at retrieval time. All candidates should be judged against one statement of intent.
For compound questions there's a wrinkle: a chunk that answers only half of "compare SSO support in Okta and Entra ID" may score mediocre against the full compound query. In that case, score per aspect and take the max per chunk rather than the mean, then fuse. You're telling the system that a candidate only needs to earn its place once.
Two production constraints to design around:
Bound the candidate pool. Reranker cost is linear in candidates. Three arms returning 200 each is not a free recall win; it's a latency incident. Retrieve deeply enough to cover the plausible matches, cap the merged pool, then rerank.
Respect the reranker's sequence limit. That example cross-encoder has a roughly 512-token context window. Feed it a 2,000-token chunk and you're scoring its opening paragraphs while citing its ending. Either chunk to fit, or retrieve small units, rerank those, and expand to a parent window after selection. The latter — small retrieval unit, larger generation unit — is usually the better default for documentation.
🚨 Production warning: A reranker trained on English web search passages is not automatically good at your Japanese runbooks or your Kubernetes error logs. Evaluate it on your corpus. It's a model, not a magic relevance oracle.
7. When More Queries Make Your Answers Worse
More retrieval is not monotonically better at the system level. Four mechanisms will find you.
Context dilution. Every extra chunk is a chance for the generator to anchor on the wrong one. Long-context position sensitivity is well documented: a relevant passage buried in the middle of a large context is not treated the same as one at the edges. Expanding from six to twenty chunks can lower answer quality while raising recall@20. Those aren't contradictory results; they're measuring different stages.
Correlated blind spots. Paraphrases generated by one model tend to share that model's vocabulary assumptions. If your corpus says named user and your model keeps saying seat, five paraphrases of seat retrieve five copies of the wrong page. A useful diagnostic: measure pairwise similarity between your query embeddings. If the "diverse" queries all land in the same neighborhood, you haven't fanned out — you've paid a toll to stand still.
Hard constraints don't rank. "Not," "excluding," "only," a version number, a region, a tenant — these are predicates, not preferences. Dense retrieval handles negation poorly, and no amount of paraphrasing enforces an ACL. Put these in filters. A fan-out that faithfully preserves a constraint in prose while omitting it from the filter is a bug with extra steps.
Expansion manufactures evidence for unanswerable questions. Ask enough variations and you will always retrieve something topically adjacent. Then the generator has material, and models are inclined to use it. Your abstention path needs a gate on the canonical intent and the reranked evidence — not on how many variants happened to vote for a chunk.
The cost side is worth writing down explicitly:
| Approach | Extra serial LLM hops | Embedding calls | ANN searches | Rerank pool |
|---|---|---|---|---|
| Baseline | 0 | 1 | 1 | k |
| Rewrite only | 1 | 1 | 1 | k |
| Fan-out only, one generation call | 1 | 1 batched | N | up to N × k |
| Rewrite + fan-out | 2 | 1 batched | N | up to N × k |
| Iterative agentic search | 1…N, sequential | variable | variable | variable |
These are structural counts assuming batching and parallel search, not benchmark results. The takeaway is that the expensive part is the serial model hop, not the vector search. Two transformation stages means two round trips before retrieval even starts.
That suggests an optimization worth stealing: run the raw-query search speculatively while the rewrite is in flight. You'll often discard the result, but on a cache miss it hides a round trip, and it gives you a fallback arm when the rewrite fails its guardrails. Filter it server-side like any other search — speculative doesn't mean unauthenticated.
8. HyDE, Decomposition, and Agentic Search Are Not the Same Thing
Three techniques get lumped into "query transformation" blog posts. They solve different problems, and conflating them leads to some genuinely strange architectures.
HyDE embeds a document, not a question
Hypothetical Document Embeddings generate a plausible passage that would answer the question, embed that passage, and retrieve against it. The intuition is distributional: a fake answer looks more like a real document than a terse question does.
That makes HyDE interesting when user language and document language are far apart — a symptom description against a technical manual. It makes HyDE dangerous when the answer is a specific fact. Ask about a timeout default and the model happily invents 30 seconds; now you're retrieving documents about 30 seconds, which may not be the truth.
Two rules if you use it: keep the literal query as a separate arm, and never pass the hypothetical passage to the generator as evidence. It's a retrieval probe. It has no provenance, and treating it as a source is how you get a citation to a document that doesn't exist.
Decomposition is for dependencies, not synonyms
"Which retention policy applies to the plan our workspace is on?" cannot be answered by three parallel paraphrases. You must resolve the plan, then retrieve its policy. That's a dependency, and it needs either a small explicit DAG or a sequential tool-calling loop.
There's a second design choice hiding here: after decomposing, do you retrieve all evidence and synthesize once, or answer each sub-question and compose the answers?
Sub-answer composition is tempting and lossy. It discards provenance at each hop, and errors compound. For most RAG systems, retrieve the union of evidence and synthesize once, keeping chunk IDs attached. Only compose sub-answers when the evidence genuinely cannot fit in one context, and accept that you've traded citation fidelity for capacity.
Agentic search is the third option
Rather than fanning out once, let the model retrieve, inspect the results, identify a gap, and search again. This has a higher ceiling on multi-hop questions and on cases where you don't yet know which vocabulary to search.
It also has an unbounded-ish cost profile. If you ship it, ship it with a maximum search count, a token budget, a wall-clock deadline, and deduplication across turns so the fifth search doesn't re-fetch the first three results.
My inclination: static multi-query for phrasing and facet coverage, agentic search for genuinely sequential questions. Raising your fan-out factor from three to twelve is a poor imitation of a model that can read a result and change its mind.
9. Evaluate at a Fixed Context Budget or You Are Fooling Yourself
This is where most query-transformation comparisons fall apart.
You run the baseline with top_k=10. You run multi-query with three arms at top_k=20, merge, and pass sixty chunks to the generator. Recall goes up. Of course it does. You changed two variables and attributed the result to one.
Fix the budgets, then vary the strategy. There are two budgets that matter:
- The candidate pool handed to the reranker.
- The final context handed to the generator.
Compare arms at identical values for both. If multi-query gets sixty candidates and a six-chunk final context, the baseline gets a sixty-candidate pool from a deeper single search and the same six-chunk final context. Now you're measuring whether diverse retrieval beats deep retrieval, which is the question you actually care about.
Label gold chunks against user intent, and freeze those labels across arms. A rewrite changing the query string must not change what counts as correct — otherwise you've built a test that rewards itself.
def recall_at_k(retrieved: list[str], gold: set[str], k: int) -> float:
return len(set(retrieved[:k]) & gold) / len(gold) if gold else 0.0
def mrr(retrieved: list[str], gold: set[str]) -> float:
for rank, chunk_id in enumerate(retrieved, start=1):
if chunk_id in gold:
return 1.0 / rank
return 0.0
Then run the ablation matrix deliberately:
| Arm | What it isolates |
|---|---|
| Baseline hybrid search | Your floor; often better than teams expect |
| Rewrite only | Does context repair explain the gains? |
| Fan-out only, no rewrite | Does vocabulary coverage explain the gains? |
| Rewrite + fan-out + RRF | Does fusion help, or just add noise? |
| Above + rerank at fixed budget | Does the extra recall actually convert? |
Report recall at the candidate stage and at the final-context stage separately, plus a relevance-weighted ordering metric (MRR or nDCG), plus end-to-end answer correctness. A run that improves recall@60 and leaves answer correctness flat is telling you something important: your bottleneck was never retrieval.
Log, per query and per arm: the original turn, the rewrite, each variant, the candidate IDs, the final context IDs, latency, and token cost. When an arm loses, you want to diff two traces, not re-run a black box.
Two evaluation traps worth naming.
LLM-as-judge rewards fluency. A confident, well-cited wrong answer often scores well. Use a judge for triage at scale, keep a human-labeled core set as ground truth, and if you do pairwise judging, swap the order — position bias is real and will manufacture a winner.
Small eval sets don't resolve small deltas. A hundred queries can show you a five-point regression. It cannot reliably resolve a one-point "win," especially when arms are compared on the same queries. Use paired per-query deltas and a bootstrap over queries rather than comparing two independently rounded percentages.
And measure the boring dimensions on the same runs: p95 latency, cost per query, and behavior on the unanswerable subset. A transformation that improves average quality while doubling tail latency and halving abstention accuracy is not an unqualified win.
10. A Routing Policy You Can Ship on a Friday
Don't apply query transformation uniformly. The queries in a real product are not a homogeneous distribution, and the expensive path should be reserved for the queries that need it.
| Query signal | Action |
|---|---|
| Pronouns, ellipsis, "and also…", topic carry-over | Contextual rewrite; don't fan out an unresolved query |
| Single lookup containing an ID, version, or exact error code | Literal query + filters; expansion usually adds noise |
| Clear concept question, jargon-heavy or multilingual corpus | Rewrite if needed, then 2–3 angled queries |
| Comparative or dependency-bearing question | Decompose; consider a bounded agentic loop |
| Exclusion, region, plan, or tenant constraint | Server-side filters; verify literals survived |
| Weak reranked evidence | Clarify or abstain — don't expand harder |
A reasonable default pipeline, in order:
authenticated request context
↓
contextual rewrite ── guardrail fails ──→ original query
↓
route: canonical lookup → 1 query
otherwise → 2–3 angled queries
↓
batched embedding → parallel, ACL-filtered hybrid search
↓
dedupe by chunk ID → RRF → bounded candidate pool
↓
cross-encoder rerank against the canonical intent
↓
budgeted context with citations → answer, clarify, or abstain
Before turning any of this on for real:
- Cache rewrites and expansions, keyed by prompt version and model version. Support-bot query distributions are skewed enough that this matters.
- Cache query embeddings too — but include tenant and ACL scope in any cache key that could serve retrieved results.
- Give every LLM hop a timeout and a fallback to the untransformed query. A rewrite service being down should degrade retrieval quality, not availability.
- Keep the original query as an arm when rewrite confidence is low.
- Shadow the new pipeline against the old one before routing traffic, and watch escalation rate and unsupported-citation rate alongside answer quality.
What I'd actually choose: for a typical product-documentation RAG system, I'd ship conservative contextual rewriting, hybrid search, and a reranker first. I'd add multi-query retrieval only where the eval set shows a vocabulary or facet failure that those three don't already fix. Three angles, one batched embedding call, one canonical rerank intent. Not ten paraphrases.
The reason is unglamorous: rewriting and reranking are easier to debug and easier to constrain. A rewrite is one inspectable string with a guardrail in front of it. A ten-way fan-out is a distribution of intentions, and when the answer is wrong you'll spend an afternoon figuring out which arm contributed the chunk that misled the generator.
Rewrite for meaning. Fan out for coverage. Rerank for evidence.
The question isn't which technique wins. It's which of your failures you're paying to fix — and your query log already knows the answer.