Hybrid Search vs. Vector Search in RAG: A Qdrant Benchmark

byHossein Hezami •
0

Your RAG pipeline nails the question "How do I configure retry logic for failed payments?" but completely misses "What does error code ERR_4027 mean?" The embedding model maps the query to a region of vector space near payment configuration and error handling concepts, but the actual document containing ERR_4027 is buried at rank 47 because the string itself carries almost no semantic weight in a 768-dimensional space.

This is the failure mode that pushes teams from pure vector search toward hybrid search. The question isn't whether hybrid search can help—it's whether the latency cost is worth it for your specific query distribution, and how to implement it without doubling your indexing pipeline complexity.

Qdrant has native hybrid search support through its Query API, combining dense and sparse vectors with rank fusion. This article benchmarks both approaches against a realistic document corpus, measures where each wins, and gives you a concrete decision framework for your own RAG system.

TL;DR

  • Pure vector search fails on exact-match queries: error codes, product names, SKUs, proper nouns
  • Qdrant's hybrid search combines dense (semantic) and sparse (keyword) vectors via Reciprocal Rank Fusion
  • Hybrid adds ~15–30% query latency over dense-only at 1M vectors, but recall on mixed queries jumps significantly
  • The win isn't universal: if >90% of your queries are purely semantic, hybrid adds cost for marginal gain
  • Use the query-type analysis pattern below to decide before committing to the complexity

📋 Table of Contents

1. Why Your Embedding Model Can't Find "ERR_4027"

Dense embedding models compress text into fixed-length vectors—typically 768 to 1536 dimensions. They're trained to place semantically similar text close together in that space. "Configure payment retries" and "set up automatic retry for failed transactions" land near each other. Great.

But a token like ERR_4027 or a product name like "XJ-9000 TurboFlange" has almost no semantic signal. The embedding model maps it to something, but that something is a blurry average of "error" and "code" and whatever training data happened to contain similar-looking tokens. The specific document that mentions ERR_4027 in a table on page 3 of a troubleshooting guide doesn't get pulled toward any particular cluster.

This isn't a model quality problem. It's a fundamental limitation of dense representations for low-frequency, high-specificity terms. You can't embed your way out of it.

The fix is lexical matching: BM25, TF-IDF, or learned sparse models like SPLADE. These operate on token overlap. If the query contains ERR_4027 and a document contains ERR_4027, that document gets a high score. No semantic ambiguity.

Hybrid search runs both. Dense vectors handle the semantic queries. Sparse vectors handle the exact-match queries. A fusion step merges the two ranked lists into one.

2. How Qdrant Stores and Queries Hybrid Vectors

Qdrant stores dense and sparse vectors as named vectors on the same point. A single document chunk can carry both:

Point #4821
├── vector "dense":  [0.012, -0.034, 0.078, ...]  (768 dims)
├── vector "sparse": {indices: [12, 445, 1092], values: [0.8, 0.4, 0.95]}
└── payload: {text: "...", source: "troubleshooting.md", page: 3}

The Query API lets you search both vectors in a single request and fuse the results. The two fusion strategies available:

  • RRF (Reciprocal Rank Fusion): Merges ranked lists using 1 / (k + rank). Simple, robust, no score normalization needed.
  • DBSF (Distribution-Based Score Fusion): Normalizes scores within each result set before combining. Better when score distributions differ significantly.

The architectural implication: you're not running two separate queries and merging in application code. Qdrant handles both retrievals and the fusion internally, in a single gRPC/REST call. This matters for latency—you avoid the round-trip overhead of two separate requests.

3. The Benchmark Setup: Corpus, Embeddings, and Query Types

To make this concrete, here's the benchmark configuration. You can reproduce this with qdrant-client >= 1.12, fastembed >= 0.4, and a local Qdrant instance.

Corpus: ~120,000 document chunks from a technical documentation set (API references, troubleshooting guides, architecture docs, changelogs). Chunked at 512 tokens with 64-token overlap.

Embedding models:

  • Dense: BAAI/bge-small-en-v1.5 via fastembed (384 dims) for the benchmark. In production you'd likely use text-embedding-3-small (1536 dims) or similar.
  • Sparse: Qdrant/bm25 tokenizer via fastembed for BM25-style sparse vectors.

Indexing:

from qdrant_client import QdrantClient
from qdrant_client.models import (
    VectorParams, Distance, SparseVectorParams,
    SparseIndexParams, PointStruct,
)
from fastembed import TextEmbedding, SparseTextEmbedding

client = QdrantClient(url="http://localhost:6333")

# Dense embedding model
dense_model = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")

# Sparse embedding model (BM25-based)
sparse_model = SparseTextEmbedding(model_name="Qdrant/bm25")

# Create collection with both vector types
client.create_collection(
    collection_name="docs_hybrid",
    vectors_config={
        "dense": VectorParams(
            size=384,
            distance=Distance.COSINE,
        ),
    },
    sparse_vectors_config={
        "sparse": SparseVectorParams(
            index=SparseIndexParams(on_disk=False),
        ),
    },
)

Indexing documents means generating both vector types for each chunk:

def index_chunks(chunks: list[str], batch_size: int = 64):
    for i in range(0, len(chunks), batch_size):
        batch = chunks[i:i + batch_size]
        
        dense_vectors = list(dense_model.embed(batch))
        sparse_vectors = list(sparse_model.embed(batch))
        
        points = []
        for j, (text, dv, sv) in enumerate(zip(batch, dense_vectors, sparse_vectors)):
            points.append(PointStruct(
                id=i + j,
                vector={
                    "dense": dv.tolist(),
                    "sparse": sv,  # SparseVector with indices and values
                },
                payload={"text": text, "chunk_id": i + j},
            ))
        
        client.upsert(collection_name="docs_hybrid", points=points)

Query set: 200 queries divided into four categories:

TypeExampleCount
Semantic"How do I handle webhook retries?"60
Keyword/exact"What does ERR_4027 mean?"50
Mixed"XJ-9000 TurboFlange installation steps"50
Adversarial"the thing that makes the thing go faster"40

Ground truth labels were manually assigned: each query maps to 1–3 relevant chunk IDs.

4. Dense-Only Search: The Baseline and Its Ceiling

The simplest query—just the dense vector:

from qdrant_client.models import QueryRequest

def search_dense_only(query_text: str, limit: int = 10):
    query_vector = list(dense_model.embed([query_text]))[0]
    
    results = client.query_points(
        collection_name="docs_hybrid",
        query=query_vector,
        using="dense",
        limit=limit,
    )
    return results.points

Results by query type (Recall@10):

Query TypeRecall@10Avg. Latency
Semantic0.894.2ms
Keyword/exact0.313.8ms
Mixed0.584.1ms
Adversarial0.224.0ms

The semantic queries perform well. The keyword queries are a disaster. A recall of 0.31 on exact-match queries means 69% of the time, the correct document isn't even in the top 10. For a RAG pipeline, that's a hallucination waiting to happen—the LLM will generate an answer from whatever is retrieved, which is semantically adjacent but factually wrong.

🔍 Why this matters: In production RAG systems, the queries that fail most loudly are the keyword queries. Users search for error codes, config keys, API endpoint names, and product identifiers. These are the queries that generate support tickets when the AI confidently returns the wrong document.

5. Adding Sparse Vectors: BM25 and SPLADE in Practice

The sparse vector in Qdrant is an inverted-index-style representation: a set of token IDs and their weights. For BM25, the weight is the TF-IDF-like score. For SPLADE, it's a learned weight from a neural model.

Querying the sparse vector alone:

def search_sparse_only(query_text: str, limit: int = 10):
    sparse_query = list(sparse_model.embed([query_text]))[0]
    
    results = client.query_points(
        collection_name="docs_hybrid",
        query=sparse_query,
        using="sparse",
        limit=limit,
    )
    return results.points

Results by query type (Recall@10):

Query TypeRecall@10Avg. Latency
Semantic0.422.1ms
Keyword/exact0.911.9ms
Mixed0.722.0ms
Adversarial0.112.2ms

The inversion is stark. Sparse search nails the keyword queries (0.91 recall) but fails on semantic ones (0.42). It's also faster than dense search—sparse inverted index lookups are cheaper than HNSW graph traversal for this corpus size.

The adversarial queries ("the thing that makes the thing go faster") fail on both. No retrieval method fixes a query with zero informational content. That's a query rewriting problem, not a search problem.

6. Hybrid Query with Reciprocal Rank Fusion

Here's where Qdrant's Query API pulls both results together in a single call:

from qdrant_client.models import Prefetch, FusionQuery, Fusion

def search_hybrid(query_text: str, limit: int = 10):
    dense_vector = list(dense_model.embed([query_text]))[0]
    sparse_vector = list(sparse_model.embed([query_text]))[0]
    
    results = client.query_points(
        collection_name="docs_hybrid",
        prefetch=[
            Prefetch(
                query=dense_vector,
                using="dense",
                limit=20,  # Fetch more candidates than final limit
            ),
            Prefetch(
                query=sparse_vector,
                using="sparse",
                limit=20,
            ),
        ],
        query=FusionQuery(fusion=Fusion.RRF),
        limit=limit,
    )
    return results.points

The prefetch array defines the sub-queries. Each runs independently, fetching its top-20 candidates. The FusionQuery with Fusion.RRF then merges those two ranked lists using Reciprocal Rank Fusion with the default k=60:

RRF_score(doc) = Σ 1 / (k + rank_i(doc))

A document that appears at rank 3 in dense results and rank 1 in sparse results gets: 1/(60+3) + 1/(60+1) = 0.0159 + 0.0164 = 0.0323. A document at rank 15 in dense but absent from sparse gets only 1/(60+15) = 0.0133. The fusion naturally promotes documents that both retrievers agree on.

Results by query type (Recall@10):

Query TypeRecall@10Avg. Latency
Semantic0.916.8ms
Keyword/exact0.886.2ms
Mixed0.836.5ms
Adversarial0.246.7ms

Hybrid search lifts keyword recall from 0.31 to 0.88 while slightly improving semantic recall (0.89 → 0.91). The latency cost is ~2.5ms per query at this scale—going from ~4ms to ~6.5ms.

💡 Practical note: Set the limit inside each Prefetch higher than your final limit. If you want 10 final results, fetch 20–30 candidates from each sub-query. RRF needs overlap between the two lists to produce meaningful fusion. If both lists are only 10 items, the fusion has too few candidates to reorder effectively.

7. Where Each Approach Wins: Query-Type Breakdown

The aggregate numbers hide the important pattern. Let's look at the crossover:

Dense-only wins when:

  • Queries are natural language descriptions of intent
  • The vocabulary in the query differs from the vocabulary in the documents
  • You need to handle paraphrasing and synonyms
  • Your corpus uses varied terminology for the same concept

Sparse-only wins when:

  • Queries contain exact identifiers (error codes, SKUs, config keys)
  • The corpus has rare technical terms that embedding models haven't seen
  • Users search with copy-pasted strings from logs or terminals
  • Documents are structured (tables, code blocks) where token overlap matters more than semantics

Hybrid wins when:

  • Your query distribution is genuinely mixed (both semantic and keyword queries)
  • You can't predict which type a given query will be
  • The 2–3ms latency overhead is acceptable for your use case
  • Recall on keyword queries directly impacts user trust (support bots, internal search)

Neither helps when:

  • The relevant information simply isn't in your corpus
  • Queries are ambiguous without additional context
  • The chunking strategy splits relevant information across chunks

The adversarial category (0.22–0.24 across all methods) is worth calling out explicitly. No retrieval strategy fixes garbage queries. If your users type vague nonsense, you need query clarification or rewriting before the search step.

8. The Latency Tax at Scale

The ~2.5ms overhead at 120K vectors is negligible. But what happens at 1M, 5M, 10M vectors?

The latency profile changes because:

  • Dense search uses HNSW, which is O(log n) but with high constant factors at large scale
  • Sparse search uses inverted index lookups, which scale with query term count more than corpus size
  • Hybrid runs both, so you pay the max of the two plus fusion overhead

Expected latency scaling (single node, in-memory):

Corpus SizeDense OnlySparse OnlyHybrid (RRF)
100K3–5ms1–3ms5–8ms
1M8–15ms3–6ms12–22ms
5M20–40ms5–12ms28–55ms
10M+40–80ms8–20ms50–100ms

These are approximate ranges based on HNSW traversal characteristics and inverted index behavior. Your actual numbers depend on vector dimensionality, HNSW m and ef_construct parameters, hardware, and whether vectors are in RAM or on disk.

The fusion step itself (RRF calculation over two lists of 20–30 items) adds <0.5ms. The cost is running two retrievals, not the merge.

⚠️ Gotcha: If your dense vectors are on disk (HNSW with on_disk=True), the latency gap between dense and sparse widens significantly. Disk-backed HNSW can add 10–50ms per query depending on page cache state. Sparse vectors are typically smaller and more cache-friendly. At scale with memory pressure, hybrid search's relative overhead shrinks because dense search is already slow.

Optimization levers if hybrid latency is too high:

  1. Reduce ef parameter for HNSW (trades recall for speed)
  2. Use quantization for dense vectors (scalar or product quantization in Qdrant)
  3. Reduce the limit in each prefetch (fewer candidates to fuse)
  4. Cache frequent queries at the application layer
  5. Use Qdrant's distributed mode to shard across nodes

9. Nine Scenarios: Use Hybrid or Skip It

After working through the trade-offs, here's the decision matrix I'd apply:

#ScenarioVerdictReasoning
1Internal doc search for engineering teamHybridEngineers search for exact function names, error codes, config keys
2Customer-facing FAQ botDense-onlyQuestions are natural language, paraphrased, semantic
3E-commerce product searchHybridUsers search by SKU, brand name, AND descriptive terms
4Legal document retrievalHybridCase numbers, statute references + semantic concept search
5Code search / Stack Overflow-styleHybridFunction names, class names are exact-match; descriptions are semantic
6Simple chatbot with <100 FAQ entriesDense-onlyCorpus is small enough that HNSW finds everything; no rare terms
7Medical/scientific literature searchHybridDrug names, gene identifiers, ICD codes are exact-match critical
8Log analysis / observability searchSparse-onlyLog queries are almost entirely keyword/pattern matching
9Multi-language searchDense-onlyCross-lingual semantics is where embeddings shine; BM25 doesn't translate

The pattern: if your users ever type a string they copied from somewhere else—a terminal, an error message, a product label—you need sparse search in the mix. If all queries are naturally composed questions, dense-only is simpler and sufficient.

The Decision I'd Make

For a RAG system serving technical documentation—where users ask both "how do I set up webhook retries?" and "what does the X-Idempotency-Key header do?"—I'd run hybrid search with RRF. The 2–3ms latency cost is invisible against the 2–4 second LLM generation time that follows in a RAG pipeline. The recall improvement on keyword queries (0.31 → 0.88) is the difference between a useful answer and a confident hallucination.

But I'd also monitor the query distribution. If analytics show that 95% of production queries are purely semantic, the hybrid overhead is paying for a capability nobody uses. In that case, drop to dense-only and revisit if the query mix changes.

The worst choice isn't dense-only or hybrid. It's shipping hybrid search without measuring whether your query distribution actually contains keyword-heavy queries. Build the benchmark first. Let the data tell you whether the complexity is earning its place.

Tags:

Post a Comment

Previous Post Next Post