Technology · Search engine
Elasticsearch
A distributed search and analytics engine built on inverted indexes. Indexing and analysis, relevance, shards and replicas, keeping it in sync, and running it well.
11 min read
Elasticsearch (and its fork OpenSearch) answers queries a database can't do well: full-text search with relevance ranking, typo tolerance, facets, and fast aggregations over huge volumes of logs and events. It's almost always a derived store, fed from a system of record.
When to reach for it
- Full-text search over products, documents, messages or tickets.
- Autocomplete and fuzzy matching.
- Faceted navigation: counts by brand, price band, category.
- Log and event analytics over time-based indexes.
Keep authoritative data (orders, balances) in a database; Elasticsearch isn't designed to be the only copy.
How indexing works
- A document arrives as JSON. Each text field goes through an analyzer: tokenize, lowercase, remove stop words, stem.
- The resulting terms go into an inverted index: term → list of document ids and positions.
- Writes are buffered and become searchable at the next refresh, by default every second. That's why it's "near real time".
- Data is stored in immutable segments, merged in the background like an LSM tree.
The mapping declares each field's type: text for analyzed full-text, keyword for exact values used in filters, sorting and aggregations, plus numbers, dates and geo points.
Relevance and queries
- Scoring uses BM25: terms that are rare across the corpus and frequent in the document score higher.
- Combine a scored query with unscored filters (status, tenant, date range); filters are cached and cheap.
- Boost fields (title over body), add function scores for freshness or popularity, and use fuzziness or edge n-grams for typos and prefixes.
Shards and replicas
An index is split into primary shards, each with replicas on other nodes. Replicas add availability and read throughput. Every search touches every shard, so too many small shards cost more than they save.
- Target shard size
- ~10–50 GB
- Search latencydepends on query and shard count
- tens of ms
- Refresh intervalraise it for heavy indexing
- 1 s default
- Heap per nodethe rest of RAM goes to the file cache
- up to ~31 GB
Keeping it in sync
Dual write from the app
- Simple
- Crash between writes leaves the index stale
Pick it when: Prototypes.
CDC or outbox into an indexer
- Every committed change arrives
- Retryable
- A pipeline to run
- Seconds of lag
Pick it when: The standard production answer.
Periodic full reindex
- Fixes drift
- Enables mapping changes
- Heavy
Pick it when: Alongside CDC, using index aliases to swap atomically.
Use index aliases: build a new index in the background, then switch the alias, so mapping changes never need downtime.
Where it bites
- Deep pagination is expensive; use
search_afterinstead of large offsets. - Mapping explosions from arbitrary user keys bloat the cluster state; use flattened fields.
- High-cardinality aggregations use a lot of memory.
Try it
Document Search Service HLD
Design full-text search over millions of documents.