What is a precision at k?
A metric for evaluating a ranked (ordered) list of items.
precision at k explained in plain English
A metric for evaluating a ranked (ordered) list of items. Precision at k identifies the fraction of the first k items in that list that are "relevant." That is: \[\text{precision at k} = \frac{\text{relevant items in first k items of the list}} {\text{k}}\] The value of k must be less than or equal to the length of the returned list. Note that the length of the returned list is not part of the calculation. Relevance is often subjective; even expert human evaluators often disagree on which items are relevant. Compare with: - average precision at k - mean average precision at k
Suppose a large language model is given the following query:
And the large language model returns the list shown in the first two columns of the following table: Relevant? | --- | Yes | Yes | No | Yes | No | Yes | Two of the first three movies are relevant, so precision at 3 is:
Example
Three of the first five movies are very funny, so precision at 5 is:
People also read
- average precision at k
A metric for summarizing a model's performance on a single prompt that generates ranked results, such as a numbered list of book recommendations.
- BERT
A model architecture for text representation.
- Character N-gram F-score
A metric to evaluate machine translation models.
- citation precision
A metric that answers the following question: What percentage of the citations in an LLM's response were actually correct and supportive?
- citation recall
A metric that answers the following question: What percentage of the source documents the LLM used to compose its response are actually cited in the response?
- cross-entropy
A generalization of Log Loss to multi-class classification problems.
- denoising
A common approach to self-supervised learning in which: 1.
- depth
The sum of the following in a neural network: - the number of hidden layers - the number of output layers, which is typically 1 - the number of any embedding layers For example, a neural network with five hidden layers and one output layer has a depth of 6.
- Embedding
A numerical representation of text, images, or other data that captures semantic meaning.
- embedding layer
A special hidden layer that trains on a high-dimensional categorical feature to gradually learn a lower dimension embedding vector.