What is a Gradient Descent?
The method by which a model gradually improves by making small adjustments after each mistake, moving toward better performance.
Gradient Descent explained in plain English
Gradient descent is the method by which a machine learning system gradually improves. It makes small adjustments after each mistake, slowly moving toward better performance — like finding the bottom of a valley by always walking downhill.
Variants of this algorithm underpin virtually all modern model training.
Analogy
Example
How is Gradient Descent used?
Common misconceptions about Gradient Descent
Gradient descent does not guarantee the best possible solution — it finds a good local minimum, and learning rate choices matter enormously.
People also read
- Backpropagation
The process that tells a neural network which internal settings caused an error and how to adjust them, working backwards through layers.
- Bayesian neural network
A probabilistic neural network that accounts for uncertainty in weights and outputs.
- embedding layer
A special hidden layer that trains on a high-dimensional categorical feature to gradually learn a lower dimension embedding vector.
- full softmax
Synonym for softmax.
- generative model
Practically speaking, a model that does either of the following: - Creates (generates) new examples from the training dataset.
- gradient boosting
A training algorithm where weak models are trained to iteratively improve the quality (reduce the loss) of a strong model.
- Inference
The phase when a trained model is actually used — taking new input and producing a prediction or response.
- input layer
The layer of a neural network that holds the feature vector.
- logistic regression
A type of regression model that predicts a probability.
- minimax loss
A loss function for generative adversarial networks, based on the cross-entropy between the distribution of generated data and real data.