What is a fraction of successes?
A metric for evaluating an ML model's generated text.
fraction of successes explained in plain English
A metric for evaluating an ML model's generated text. The fraction of successes is the number of "successful" generated text outputs divided by the total number of generated text outputs. For example, if a large language model generated 10 blocks of code, five of which were successful, then the fraction of successes would be 50%. Although fraction of successes is broadly useful throughout statistics, within ML, this metric is primarily useful for measuring verifiable tasks like code generation or math problems.
Example
Practitioners refer to fraction of successes when building, training, or evaluating machine learning systems. It appears in research papers, product documentation, and technical discussions about AI capabilities and limitations.
People also read
- cross-entropy
A generalization of Log Loss to multi-class classification problems.
- embedding layer
A special hidden layer that trains on a high-dimensional categorical feature to gradually learn a lower dimension embedding vector.
- encoder
In general, any ML system that converts from a raw, sparse, or external representation into a more processed, denser, or more internal representation.
- Inference
The phase when a trained model is actually used — taking new input and producing a prediction or response.
- language model
A model that estimates the probability of a token or sequence of tokens occurring in a longer sequence of tokens.
- perplexity
One measure of how well a model is accomplishing its task.
- AUC
A number between 0.
- average precision at k
A metric for summarizing a model's performance on a single prompt that generates ranked results, such as a numbered list of book recommendations.
- Backpropagation
The process that tells a neural network which internal settings caused an error and how to adjust them, working backwards through layers.
- bag of words
A representation of the words in a phrase or passage, irrespective of order.