What is Gradient Descent?
The method by which a model gradually improves by making small adjustments after each mistake, moving toward better performance.
Gradient Descent explained in plain English
Gradient descent is the method by which a machine learning system gradually improves. It makes small adjustments after each mistake, slowly moving toward better performance — like finding the bottom of a valley by always walking downhill.
Variants of this algorithm underpin virtually all modern model training.
Analogy
Gradient descent is like descending a foggy hill by always feeling which direction slopes downward and taking one careful step at a time. You cannot see the bottom, but each step brings you closer.
Example
When engineers train a new LLM over weeks on thousands of GPUs, gradient descent is the optimisation process adjusting billions of parameters.
How is Gradient Descent used?
Common misconceptions about Gradient Descent
Gradient descent does not guarantee the best possible solution — it finds a good local minimum, and learning rate choices matter enormously.
People also read
- Backpropagation
The process that tells a neural network which internal settings caused an error and how to adjust them, working backwards through layers.
- Bayesian neural network
A type of neural network that uses Bayesian inference to make predictions
- embedding layer
A layer in a neural network that converts input data into a dense vector representation
- full softmax
A mathematical function used in machine learning to normalize a set of values into probabilities
- generative model
A type of artificial intelligence model that generates new data samples based on a given dataset
- gradient boosting
A machine learning technique that combines multiple weak models to create a strong predictive model
- Inference
The phase when a trained model is actually used — taking new input and producing a prediction or response.
- input layer
The first layer of a neural network that receives input data
- logistic regression
A statistical method for predicting the outcome of a categorical variable based on one or more predictor variables
- minimax loss
A loss function used in machine learning to train models, especially in game-playing AI and reinforcement learning