What is perplexity?
A measure of how well a language model predicts a test set
perplexity explained in plain English
Perplexity is a way to evaluate the performance of a language model, which is a type of artificial intelligence that generates human-like text. It measures how well the model can predict the next word in a sentence, given the context of the previous words. A lower perplexity score indicates that the model is better at predicting the next word.
Analogy
Think of perplexity like a game of guessing the next word in a sentence. Imagine you're trying to guess the next word in a story, and you have a few options to choose from. If you're really good at guessing, you'll have a low perplexity score. But if you're not very good, you'll have a high perplexity score, because there are many possible words that could come next.
Example
For instance, a language model designed to generate product descriptions might have a perplexity score of 100 on a test set of descriptions. This means that, on average, the model is as uncertain as if it had 100 possible next words to choose from. If the model is improved and its perplexity score drops to 50, it means the model is now as uncertain as if it had only 50 possible next words to choose from.
How is perplexity used?
Perplexity is used to compare the performance of different language models, and to evaluate how well a model is doing on a particular task. For example, a model with a low perplexity score on a test set of news articles might be good at generating news-like text.
Common misconceptions about perplexity
History
The concept of perplexity was first introduced in the 1950s, in the context of information theory. It has since been widely adopted in the field of natural language processing, as a way to evaluate the performance of language models.
People also read
- cross-entropy
A measure of difference between predicted and actual outcomes
- embedding layer
A layer in a neural network that converts input data into a dense vector representation
- encoder
A component of a neural network that converts input data into a more useful form
- fraction of successes
A measure of the number of successful outcomes in a series of trials
- Inference
The phase when a trained model is actually used — taking new input and producing a prediction or response.
- language model
A computer program that understands and generates human-like language
- AUC
A measure of a model's ability to distinguish between positive and negative classes
- average precision at k
A measure of the accuracy of a model's top k predictions
- Backpropagation
The process that tells a neural network which internal settings caused an error and how to adjust them, working backwards through layers.
- bag of words
A representation of text as a collection of individual words, ignoring grammar and word order