Text Analysis

Before any scoring, a search engine turns raw text into terms. A query matches a document only when both produce the exact same term.

The TF-IDF and BM25 demos only lowercase and split on spaces, so Shoes never matches shoe,. Real engines (Lucene, Elasticsearch, OpenSearch) run both documents and queries through the same chain of steps, called an analyzer:

Tokenizerun, fast! → run fast
→
LowercaseRUN → run
→
Fold accentscafé → cafe
→
Stop wordsthe cat → cat
→
Synonymsshoes → +sneakers
→
Stem / Lemmatizerunning → run

The order matters: each step only sees what the previous one produced.

1. Settings

Tokenization

Standard also splits on punctuation.

Learn more

Whitespace keeps punctuation attached, so great! won't match great. Standard splits on anything that isn't a letter or a digit, but keeps apostrophes inside words (didn't). Hyphens split too: e-mail becomes e and mail.

Normalization

Café → cafe

Learn more

Users rarely type capitals or accents in a search box, so both sides are normalized. The cost: distinctions disappear (US the country becomes us the pronoun; in French, pâte and pâté both become pate).

Stop words

Common words that carry little meaning.

Learn more

The default list is Lucene's English stop words. Removing them shrinks the index and keeps scores focused on meaningful words, but can destroy some queries: to be or not to be becomes empty, and The Who becomes just who. With BM25's IDF already lowering common words, many modern setups keep them.

Synonyms (query only)

a, b equivalent · a => b one-way

Learn more

Here synonyms are added to the query only (query-time expansion): the index stays small and rules can change without reindexing. Expanding at index time instead makes queries faster but needs a reindex after each change.

Rules run before stemming, so they must match the unstemmed word: the rule shoes, sneakers doesn't fire for the query shoe.

Stemming / Lemmatization

Stemmer: suffix rules. Lemmatizer: dictionary lookup.

Learn more

The Porter stemmer chops suffixes with rules. It's fast and works on any word, but produces non-words (universities → univers), merges unrelated words (news → new), and misses irregular forms (ran stays ran).

A lemmatizer looks up the dictionary form: ran → run, better → good. It's accurate, but only for words it knows. This demo uses a tiny dictionary (about 150 words), so unknown words pass through unchanged.

Neither turns runners into run: see document 2 below.

2. Pipeline Explorer

Watch your text change at each step. Disabled steps are greyed out.

3. Query Matching

Which query words does each document contain? Naive splitting (as in the TF-IDF and BM25 demos) vs. the pipeline above. Documents go through the same pipeline as the query, except synonyms.