LESSON 14.3
Running a search service
Shards and replicas, reliable index updates through CDC or an outbox, refresh delay, and rebuilding a derived index without losing writes.
7 min read
A search index is optimized for retrieval, not for being the only record of an order, file or document. Keep the database as the source of truth and treat search as a derived store. That choice makes stale results explainable and a damaged index recoverable.
Shards divide, replicas copy
An Elasticsearch-style index splits documents across primary shards, often by hashing document ids. A query coordinator asks relevant shards for candidates and merges their top results.
Replicas copy partitions. A search uses one copy of each relevant shard, choosing a primary or replica. Replicas add availability and read capacity, but also storage and replication work. Place copies in separate failure domains.
- Six primary shardsAssuming an even distribution
- About 50 GB each
- One replica per primary
- 12 shard copies total
- Stored index bytesBefore merges, recovery space and headroom
- About 600 GB
- One unrestricted queryOne copy of each primary partition
- Six shard searches
More shards add coordination and memory overhead; too few can limit growth or slow recovery. Size using measured load, index size and recovery time. Tenant-based routing narrows query fan-out but can concentrate a hot tenant on one shard.
Keep the index in sync
Writing the database and then calling the search engine leaves a failure window. The database commits, the process crashes, and the index never hears about it. Reversing the calls can index data that the database never commits.
Use CDC to read committed changes from the database log, or write an outbox row in the same transaction as the source record and relay it later. The indexer transforms records into search documents and sends bounded batches to the engine.
Retries use the source record's stable id. Include a source version or ordered change position so a delayed update can't overwrite a newer value. Deletes need events too; retain tombstones or equivalent version state long enough to reject delayed older updates. Monitor backlog age, failed batches and dead-letter entries rather than quietly skipping bad records.
Near-real-time visibility
Accepting an indexing write and making it searchable are different steps. A refresh exposes recent writes to search. A roughly one-second refresh interval is a common Elasticsearch example, but deployment defaults and settings vary. Elastic's refresh documentation explains the visibility controls and the cost of forcing refreshes.
Visibility delay includes the pipeline plus refresh. An 8-second backlog isn't fixed by a 1-second refresh. For read-your-own-write behavior, show the source record or wait for that version to become searchable. Forcing every write to refresh reduces indexing efficiency.
Rebuild without losing changes
New analyzers, corrupt data or a changed document shape may require a fresh index. Build beside the live one rather than clearing it in place:
- Create a new versioned index with the desired mappings and analyzers.
- Capture a change-stream position and backfill from a consistent source snapshot, keeping changes from the matching boundary available for replay.
- Replay later updates and deletes with version checks until the new index catches up; don't let a late snapshot row overwrite a newer event.
- Validate counts, sampled records, permissions and representative queries, then switch a query alias to the new index.
- Keep feeding the new index and retain the old one briefly. A rollback requires keeping it current or catching it up before switching back.
Try it
Document Search Service HLD
Run a searchable derived store with a measurable freshness budget and a safe rebuild path.