What is Attention?
A mechanism that lets a model focus on the most relevant parts of its input when producing an output, weighting what matters most in context.
Attention explained in plain English
Attention is a mechanism that lets a model focus on the most relevant parts of its input when producing an output. Instead of treating every word or pixel equally, it weighs what matters most in context.
It solved a major bottleneck in processing long sequences and enabled the Transformer architecture.
Analogy
Attention is like reading a sentence and instinctively emphasising the words that carry the meaning, while barely registering filler words. Your focus shifts depending on what the sentence is actually about.
Example
When translating "The bank by the river" versus "The bank approved the loan," an attention-based model focuses on different words to resolve ambiguity.
How is Attention used?
Attention is the foundation of Transformers — the architecture behind GPT, Claude, and most modern language models. It also helps translation systems align words across languages.
Common misconceptions about Attention
Attention is not human-like focus or consciousness — it is a mathematical weighting scheme over inputs.
People also read
- auto-regressive model
A type of machine learning model that predicts future values based on past values
- autoencoder
A type of artificial neural network that learns to compress and reconstruct data
- depth
The number of layers in a neural network
- embedding layer
A layer in a neural network that converts input data into a dense vector representation
- embedding vector
A mathematical representation of an object, like a word or image, as a point in a high-dimensional space
- generative AI
A type of artificial intelligence that generates new content, such as images, videos, or text, based on existing data.
- Long Short-Term Memory
A type of artificial neural network used for processing sequential data
- mixture of experts
A machine learning model that combines the predictions of multiple expert models to improve overall performance
- Neural Architecture Search
A method of automatically designing neural network architectures
- pooling
A technique used in neural networks to reduce the spatial dimensions of data