What is a recall at k?
A metric for evaluating systems that output a ranked (ordered) list of items.
recall at k explained in plain English
A metric for evaluating systems that output a ranked (ordered) list of items. Recall at k identifies the fraction of relevant items in the first k items in that list out of the total number of relevant items returned. \[\text{recall at k} = \frac{\text{relevant items in first k items of the list}} {\text{total number of relevant items in the list}}\] Contrast with precision at k.
Suppose a large language model is given the following query:
And the large language model returns the list shown in the first two columns: Relevant? | --- | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | No | Yes | Eight of the movies in the preceding list are very funny, so they are "relevant items in the list." Therefore, 8 will be the denominator in all the calculations of recall at k. What about the numerator? Well, 3 of the first 4 items are relevant, so recall at 4 is:
Example
7 of the first 8 movies are very funny, so recall at 8 is:
People also read
- average precision at k
A metric for summarizing a model's performance on a single prompt that generates ranked results, such as a numbered list of book recommendations.
- BERT
A model architecture for text representation.
- Character N-gram F-score
A metric to evaluate machine translation models.
- citation precision
A metric that answers the following question: What percentage of the citations in an LLM's response were actually correct and supportive?
- citation recall
A metric that answers the following question: What percentage of the source documents the LLM used to compose its response are actually cited in the response?
- cross-entropy
A generalization of Log Loss to multi-class classification problems.
- denoising
A common approach to self-supervised learning in which: 1.
- depth
The sum of the following in a neural network: - the number of hidden layers - the number of output layers, which is typically 1 - the number of any embedding layers For example, a neural network with five hidden layers and one output layer has a depth of 6.
- Embedding
A numerical representation of text, images, or other data that captures semantic meaning.
- embedding layer
A special hidden layer that trains on a high-dimensional categorical feature to gradually learn a lower dimension embedding vector.