Evaluation
Is BM25 better than vector search? Did tweaking the analyzer help? To answer, you need a number. Search teams build test sets: queries plus relevance judgments (which documents are good answers), then score each ranking with metrics.
Each metric captures a different idea of "good". Judge the results below, reorder them, and watch the metrics react.
how to train a puppy1. Judge the Results
Query: how to train a puppy. Set a grade for each result, and move results up or down.
Learn more
Judgments usually come from human annotators following guidelines, or from user behavior (clicks, purchases). Graded judgments (0/1/2) are richer than yes/no, but only nDCG uses the grades: the other metrics just ask "relevant or not" (grade ≥ 1).
Knowing what the engine missed is the hard part: nobody can read a whole collection. In practice, judges review the top results of several systems combined (called pooling).
2. Metrics
Learn more
Precision@k: what share of the top k is relevant? Simple, but blind to order: a relevant result at rank 1 or rank k counts the same.
Recall@k: what share of all relevant documents made it into the top k? It matters when users need everything (legal search, research), less for a web search where one good answer is enough.
Reciprocal Rank: 1 / rank of the first relevant result. It models a user who stops at the first good answer (question answering, navigation). Averaged over many queries, it's called MRR.
nDCG@k uses the grades and the order: each result brings a gain (2^grade − 1) divided by a discount that grows with rank (log2(rank + 1)). Dividing by the score of the ideal ranking gives a value between 0 and 1. It's the most common metric for ranked search.