LESSON 14.4
Vector and hybrid search
Embeddings for searching by meaning, approximate nearest-neighbor indexes like HNSW and IVF, combining keyword and vector results, and reranking the top few.
6 min read
The inverted index finds documents that share words with the query. That works well for names, codes and exact phrases, and badly when people describe what they want in different words than the document uses. Vector search fills that gap.
Embeddings
An embedding model turns a piece of text (or an image) into a vector of a few hundred to a few thousand numbers. Texts with similar meaning land close together, measured by cosine similarity or dot product. "How do I get my money back?" ends up near "Refund policy" even though they share no words.
To search, embed every document (or every chunk of a long document) once at indexing time, embed the query at search time, and find the nearest vectors.
- 10 million chunks × 768 dimensions × 4 bytes
- about 31 GB of raw vectors
- Same with 8-bit quantization
- about 8 GB
- Brute-force query over 10 million
- billions of multiply-adds - far too slow per query
Approximate nearest neighbors
Exact search compares the query with every vector. ANN indexes look at a small fraction and still find almost all of the true neighbors:
HNSW (graph)
- Fast, high recall
- Supports inserts
- Memory hungry
- Deletes are awkward
Pick it when: The default in most vector databases and pgvector.
IVF (clusters)
- Smaller, faster to build
- Pairs well with compression
- Recall depends on how many clusters you probe
Pick it when: Very large collections, often with product quantization.
Brute force
- Exact
- No index
- Linear cost
Pick it when: Under about 100,000 vectors, or after a strict filter leaves few candidates.
HNSW builds a layered graph: each vector links to a few close neighbors, with sparse long-range links on the upper layers. A query starts at the top, greedily walks toward closer nodes, and drops a layer at a time, like a skip list in many dimensions.
Filters complicate things
Real queries add filters: this tenant only, in stock, newer than a date. Filtering after the ANN search can leave too few results; filtering before it may leave the graph disconnected. Vector databases handle this with filtered traversal or per-partition indexes. For strict tenant isolation, a separate index or partition per large tenant is often simplest.
Hybrid search and reranking
Neither method wins everywhere. Keyword search is precise for exact terms, product codes and rare names; vectors understand paraphrase. Production systems usually run both:
- Retrieve a few hundred candidates from each index.
- Fuse the lists, commonly with reciprocal rank fusion, which works on ranks rather than incompatible scores.
- Rerank the top 50 or so with a slower, more accurate model (a cross-encoder) that reads the query and each document together.
This retrieve-then-rerank shape is also the retrieval step behind most question-answering systems built on language models.
Try it
Document Search Service HLD
Add semantic search to a document search service.