TF-IDF
Not all words matter equally. TF-IDF (Term Frequency × Inverse Document Frequency) gives each word of a document a weight: high when the word is frequent in this document but rare in the others.
1. Documents
Words are only lowercased and split on spaces, to keep the math visible. See Text Analysis for how real engines do it.
2. Term Weights
The darker the cell, the more characteristic the word is of that document.
Learn more
TF = occurrences of the word in the document ÷ number of words in the document. Dividing by the length keeps long documents from winning just by being long.
IDF = log(N / DF), where N is the number of documents and DF the number of documents containing the word. A word present everywhere gets log(1) = 0: it can't tell documents apart. The log keeps rare words from getting absurdly large weights.
Notice the: it appears twice in D1, yet weighs less than cat or mat, because it's also in D2. And cat and cats are unrelated words here, since nothing reduces them to a common form.
3. Ranking for a Query
A document's score is the sum of the weights of the query words it contains.
Learn more
TF-IDF has two weaknesses, both fixed by BM25: a word repeated 10 times counts 10 times more than once (no saturation), and dividing by the full length over-penalizes long documents.