Elasticsearch and hybrid search

Quick definitions for jargon that recurs below; skip if you already know Elasticsearch/Lucene.

• IR (Information Retrieval) — the field concerned with finding relevant documents in a large collection in response to a query. Search engines, recommender retrieval, and RAG retrieval are all IR systems.

• Apache Lucene — the open-source Java library that implements the actual indexing and search algorithms (inverted index, scoring, segment merging). Elasticsearch and OpenSearch are distributed systems built on top of it, not reimplementations of it.

• Corpus — the full set of documents being indexed/searched.

• Index — the logical collection of documents you query in Elasticsearch, e.g. “products”.

• Shard — an index is split into shards so it can scale across machines; each shard is an independent partition of the data, and physically is one Lucene index.

• Lucene index — the on-disk index structures (inverted index, stored fields) that Lucene maintains for one shard. “Shard” is Elasticsearch’s name for it; “Lucene index” is what it actually is underneath.

• Segment — inside one Lucene index, documents live in immutable segments: writes create new segments, nothing is edited in place, and segments are merged into fewer, larger ones over time.

• BM25 — the standard lexical relevance-scoring function (term frequency, inverse document frequency, length normalization) used by Lucene/Elasticsearch.

• ANN (Approximate Nearest Neighbour) search — finding the vectors closest to a query vector without comparing against every stored vector, trading a little recall for large speed-ups.

• HNSW (Hierarchical Navigable Small World) — the graph-based ANN index used by Lucene and most vector databases.

• pgvector — a Postgres extension that adds a vector column type, distance operators, and ANN indexes to an ordinary Postgres database.

• Pinecone, Qdrant, Weaviate, Milvus, Chroma, LanceDB — purpose-built vector databases (Pinecone is managed-only; the others can be self-hosted).

Nesting, top to bottom: index → shards → (each shard = one Lucene index) → segments. “Corpus” cuts across all of it — it just means “all your documents.”

Retrieval-quality metrics used in the evaluation section below — see the Retrieval evaluation metrics subtopic for the full definitions, formulas, and a worked example:

• Recall@k — of all documents actually relevant to a query, the fraction that appear anywhere in the top-k retrieved results.
• MRR (Mean Reciprocal Rank) — averaged across queries, 1 / rank of the first relevant result; sensitive only to how high the first relevant hit lands.
• nDCG@10 (Normalized Discounted Cumulative Gain at 10) — scores the top 10 results, giving more credit the higher a relevant document ranks, normalized to [0, 1] so scores are comparable across queries.

• An index is split into shards; each shard is an independent Lucene index and can have replicas. Sharding is how the corpus scales horizontally; the shard count is largely fixed at creation.
• A shard is made of immutable segments. New documents go into an in-memory buffer, flushed (“refresh”) to a new small segment roughly every second — this is why ES is near-real-time, not real-time: a document is searchable a refresh after indexing, not instantly.
• Segments are periodically merged in the background into fewer, larger ones; deletes are tombstones applied at merge time.
• Durability is a per-shard translog (write-ahead log) fsynced on each request (by default); “flush” turns buffered data into a committed Lucene segment.
• A query fans out to one copy of every shard (the query then fetch phases: each shard returns top-k doc ids + scores, the coordinator merges, then fetches the actual documents for the global top-k).

There are really two decisions here: whether vectors belong in your search engine at all, and if not, which kind of vector store to run.

Search engine or vector database

Lean Elasticsearch when:

• You need strong lexical search (BM25), structured filtering, aggregations, faceting, geo-queries, and full-text features — and vectors are an addition, not the whole story.
• You already operate it and have the ops muscle.
• You want lexical + dense + filters resolved in one query, one system, with one consistency model.

Lean a dedicated vector DB (Qdrant, Weaviate, Milvus, LanceDB, Chroma, Pinecone, pgvector) when:

• Vectors are the primary access path at large scale and you want purpose-built ANN (better memory/latency, quantization options, GPU indexing).
• You want tight coupling with existing infra (see pgvector below).
• Simpler operational surface than a full search cluster.

ES’s vector support (HNSW via Lucene) is solid now, so the decision is less “can it” and more “is the rest of ES’s feature set worth its operational weight for your use case”.

Which vector database — pgvector vs managed vs self-hosted

Option	What it is	Strengths	Costs
pgvector	Postgres extension adding vector similarity search inside an existing Postgres database	No new infrastructure if you already run Postgres; transactional consistency between rows and their embeddings (they can’t drift out of sync)	A general-purpose relational database with a vector extension bolted on; very large vector counts may outgrow what it handles gracefully
Managed (Pinecone)	Fully managed vector service	Least ops burden; purpose-built indexing tuned for this workload	Ongoing cost, vendor lock-in
Self-hosted (Qdrant, Weaviate, Milvus, Chroma)	Purpose-built vector databases you run yourself	More advanced indexing options, built-in hybrid search (vector similarity plus keyword/metadata filtering), control and cost predictability	Real operational responsibility

The deciding factors, in order of how often they settle it:

1. Existing infrastructure — does adding a new system outweigh pgvector’s simplicity?
2. Expected scale — vector count and query rate; very large collections favour purpose-built indexes.
3. Operational appetite — managed trades money and lock-in for less ops; self-hosted the reverse.
4. Specific feature needs — hybrid search, advanced filtering, quantization, multi-tenancy may only be well supported by a purpose-built option (or by Elasticsearch itself, per above).

Hybrid search runs lexical (BM25 / sparse) and dense retrieval and combines the results. They fail in uncorrelated ways — BM25 misses paraphrases, dense misses exact identifiers and rare terms — so the union recovers relevant documents that either alone would drop, and documents ranked highly by both are strong signals. Empirically hybrid beats both on almost every benchmark (BEIR and internal evals alike), typically by a meaningful margin on recall, and the re-ranker on top then cleans up precision. The cost is running two retrieval systems and a fusion step.

Weighted score combination (a.k.a. convex / linear combination):

score(d) = alpha * norm(bm25_score(d)) + (1 - alpha) * norm(dense_score(d))

Requires normalizing the two score scales first (min-max, or z-score) because BM25 scores are unbounded and cosine is [-1, 1]. Sensitive to the normalization choice and to alpha, and score distributions drift as the corpus changes, so it needs re-tuning.

Reciprocal Rank Fusion (RRF) ignores scores entirely and uses rank:

score(d) = sum over each result list L of:  1 / (k + rank_L(d))      (k typically 60)

A document near the top of either list gets a large contribution; k damps the influence of low ranks. RRF needs no score normalization, no per-corpus tuning, and is robust — which is why it is the default in Elasticsearch, OpenSearch, and most RAG frameworks. Weighted combination can edge it out if you invest in tuning and keep the normalization honest.

The filter (e.g. tenant_id = 42 AND status = "published") must apply to both paths consistently, and the pitfalls differ:

• Lexical path — filtering is native and cheap; a filter clause in a boolean query narrows the postings scan with no scoring impact.
• Dense path — filtered ANN has the recall hazards from the vector-index subtopic (a very selective filter can starve the HNSW traversal). ES applies the filter during the vector search; if the filtered set is tiny it can fall back to exact search.
• Fusion — do the filter before fusion on each list, so both lists are already restricted; fusing then filtering can leave you with too few results.

Test with realistically selective filters, not just match_all.

• IDF is computed per shard by default — a term’s rarity is estimated from that shard’s documents, so identical documents in different shards can score differently, and small shards give noisy IDF. Mitigation: dfs_query_then_fetch (a preliminary round that gathers global term statistics) for correctness-critical scoring, at a latency cost.
• Top-k is approximate across shards — each shard returns its local top-k; a document ranked k+1 on its shard but globally top-10 is lost. Mitigation: request a larger per-shard size, or fewer shards.
• Relevance variance from uneven document routing; use routing keys deliberately if a tenant should live on one shard.
• Over-sharding wastes resources (each shard has fixed overhead) and under-sharding limits parallelism and future growth — pick shard count from target shard size (~10–50 GB) and expected growth, because you cannot easily change it later.

• Build a labeled set from real query logs: sample queries, have humans (or a strong LLM with spot-checks) mark which documents are relevant.
• Measure the BM25 baseline — recall@k, nDCG@10, MRR — and bucket by query type (keyword-y vs natural-language, head vs tail).
• Add the dense path and hybrid fusion, re-measure on the same set. Expect the gains to concentrate in the natural-language / paraphrase buckets and be near-zero (or slightly negative) on exact-identifier queries.
• Weigh the cost — an embedding model to host, a vector index to build and keep fresh, higher query latency and infra spend — against the measured recall lift on the query mix you actually serve.
• Check the downstream metric — if this feeds a RAG system, the number that matters is end-to-end answer quality, not retrieval nDCG in isolation.