LESSON 14.2
Relevance and search features
TF-IDF and BM25 intuition, exact filters and facets, prefix suggestions, and bounded fuzzy matching for misspelled queries.
6 min read
Finding documents that contain the words is only the beginning. A search for "distributed cache" should put a useful design guide ahead of a long log file that happens to repeat "cache" a thousand times. Relevance scores order the candidates; filters and suggestions help users say what they meant.
Why some matches rank higher
TF-IDF combines term frequency with inverse document frequency. A document mentioning the query term more often may be about it; a term that appears in very few documents is more informative than one appearing everywhere. Length normalization stops long documents winning just because they contain more words.
BM25 uses those ideas with a saturating term-frequency contribution and an adjustment for document length. Going from one mention to five helps; going from 100 to 104 helps much less. It's a strong starting point for lexical ranking, rather than a rule that the shortest or most repetitive document always wins. Elastic's ranking guide describes this default approach.
- cache appears in 5,000 documentsUseful, but not very selective
- A common term
- consistent-hashing appears in 20Usually contributes more IDF weight
- A rare term
- Term occurs 1 time versus 5
- Frequency can help
- Term occurs 100 times versus 104BM25 limits repetition's benefit
- Diminishing returns
Weight title matches more than body matches where useful. Phrase matches, recency or popularity can add signals, but shouldn't bury textual relevance. Evaluate real queries against judged useful results; a plausible formula isn't evidence that users find things.
Filters don't need a relevance score
Filters restrict eligibility: tenant_id = 7, language = en, price < 100. Store those values in structured fields so the engine can apply exact conditions efficiently. They shouldn't behave like stemmed prose. Every candidate must satisfy access restrictions regardless of its relevance score.
Facets aggregate the matching set, such as "Books (240), Videos (85)" or counts by brand. Define whether counts include the current filters; for some interfaces, a facet excludes its own selected filter so users can see alternatives. Limit expensive high-cardinality aggregations, especially on unbounded user input.
Typeahead: matching before the word is finished
Suggestions answer prefixes such as dis, often from a smaller corpus of titles or popular queries rather than every body field.
Prefix trie
- Follows one prefix path
- Can keep popular completions at nodes
- Memory and update costs
- Needs an explicit ranking strategy
Pick it when: A dedicated suggestion vocabulary with fast prefix lookups.
Edge n-grams
- Uses the search index
- Works with field scoring and filters
- Larger index
- Must tune minimum and maximum lengths
Pick it when: Autocomplete integrated with existing document search.
Edge n-grams index leading fragments: cache might produce ca, cac, cach, cache. Analyze the user's typed prefix compatibly, without expanding it into every fragment again unnecessarily. Debounce requests, perhaps around 150 ms, set a minimum input length, and discard stale responses that arrive after a newer query. These are starting values to tune against the product's latency budget.
Fuzzy matching
Fuzzy search tolerates a small edit distance: insertions, deletions or substitutions, sometimes adjacent transpositions. cahce can suggest cache when transpositions count as one edit. It widens matching, which costs CPU and can introduce irrelevant results.
Prefer exact and prefix matches first. Bound edit distance, require a stable prefix where useful, and cap the number of expanded terms. Don't apply fuzzy rules to identifiers or permissions. Synonyms solve a different problem: "TV" and "television" aren't typos, so they need explicit domain vocabulary rather than a larger edit budget.
Try it
Search Autocomplete HLD
Design fast, ranked prefix suggestions with bounded typo tolerance.