What is an auxiliary loss?
A loss function—used in conjunction with a neural network model's main loss function—that helps accelerate training during the early iterations when weights are randomly initialized.
auxiliary loss explained in plain English
A loss function—used in conjunction with a neural network model's main loss function—that helps accelerate training during the early iterations when weights are randomly initialized. Auxiliary loss functions push effective gradients to the earlier layers. This facilitates convergence during training by combating the vanishing gradient problem.
Example
Practitioners refer to auxiliary loss when building, training, or evaluating machine learning systems. It appears in research papers, product documentation, and technical discussions about AI capabilities and limitations.
People also read
- Backpropagation
The process that tells a neural network which internal settings caused an error and how to adjust them, working backwards through layers.
- Bayesian neural network
A probabilistic neural network that accounts for uncertainty in weights and outputs.
- convex function
A function in which the region above the graph of the function is a convex set.
- convolution
In mathematics, casually speaking, a mixture of two functions.
- convolutional filter
One of the two actors in a convolutional operation.
- convolutional layer
A layer of a deep neural network in which a convolutional filter passes along an input matrix.
- convolutional operation
The following two-step mathematical operation: 1.
- embedding layer
A special hidden layer that trains on a high-dimensional categorical feature to gradually learn a lower dimension embedding vector.
- embedding vector
Broadly speaking, an array of floating-point numbers taken from any hidden layer that describe the inputs to that hidden layer.
- full softmax
Synonym for softmax.