precision at k

A metric for evaluating a ranked (ordered) list of items. Precision at k identifies the fraction of the first k items in that list that are "relevant." That is: \[\text{precision at k} = \frac{\text{relevant items in first k items of the list}} {\text{k}}\] The value of k must be less than or equal to the length of the returned list. Note that the length of the returned list is not part of the calculation. Relevance is often subjective; even expert human evaluators often disagree on which items are relevant. Compare with: - average precision at k - mean average precision at k

Suppose a large language model is given the following query:

And the large language model returns the list shown in the first two columns of the following table: Relevant? | --- | Yes | Yes | No | Yes | No | Yes | Two of the first three movies are relevant, so precision at 3 is:

Three of the first five movies are very funny, so precision at 5 is: