activation function

A function that enables neural networks to learn nonlinear (complex) relationships between features and the label. Popular activation functions include: - ReLU - Sigmoid The plots of activation functions are never single straight lines. For example, the plot of the ReLU activation function consists of two straight lines: A plot of the sigmoid activation function looks as follows:

In a neural network, activation functions manipulate the weighted sum of all the inputs to a neuron. To calculate a weighted sum, the neuron adds up the products of the relevant values and weights. For example, suppose the relevant input to a neuron consists of the following:

Suppose the designer of this neural network chooses the sigmoid function to be the activation function. In that case, the neuron calculates the sigmoid of -2.0, which is approximately 0.12. Therefore, the neuron passes 0.12 (rather than -2.0) to the next layer in the neural network. The following figure illustrates the relevant part of the process: --- See Neural networks: Activation functions in Machine Learning Crash Course for more information.

Practitioners refer to activation function when building, training, or evaluating machine learning systems. It appears in research papers, product documentation, and technical discussions about AI capabilities and limitations.