Introduction
The activation function turns a neuron's weighted sum into its output. Its shape decides what the network can represent and how easily it can be trained. This note covers the four common choices.
Main idea
The four common activations are the binary step, piecewise linear, sigmoid $f(v)=\frac{1}{1+\exp(-av)}$ and ReLU $f(v)=\max(0,v)$. All four are nonlinear. The step gives a hard yes or no, the sigmoid is a smooth version of it, and ReLU passes positive values and blocks negative ones.
Overview
Every activation maps one number to one number, applied to each neuron separately:
The four choices differ in two things: the range of the output, and whether the function is smooth (has a slope everywhere that training can use).
Step 1: four activation functions

Binary (step):
- output 1 when , else 0
- a hard yes or no, which is the neuron as a pure linear classifier
- flat everywhere except the jump, so its slope is zero almost everywhere
Piecewise linear:
- 0 below a lower limit, 1 above an upper limit, and a straight ramp in between (the limits in the figure are only an example)
- a softened step, linear only inside the ramp
Sigmoid:
- a smooth S-curve from 0 to 1, passing at
- sets the steepness: a larger makes it closer to the step
- for large positive , so , and for large negative , so
- it has a slope everywhere, which gradient-based training needs
ReLU (rectified linear unit):
- passes positive values unchanged and sets negative values to 0
- the output has no upper limit
- the simplest nonlinear function, and very cheap to compute
- it is linear on each side of 0, but the corner at 0 makes it nonlinear overall, since in general
Step 2: comparing them
| Function | Output range | Smooth | Slope |
|---|---|---|---|
| binary | no | 0 except at the jump | |
| piecewise linear | no, corners at the limits | constant on the ramp, 0 outside | |
| sigmoid | yes | largest at , near 0 for large | |
| ReLU | no, one corner at 0 | 1 for , 0 for | |
| linear, | all real numbers | yes | 1 everywhere |
The linear function is not one of the four. It is only used at the output, when the network must produce any real number, because a hidden layer with it collapses (Multilayer Perceptron Structure).
What the slope means for training:
- training changes the weights in proportion to the slope , shown in Backpropagation
- where the slope is 0, the neuron does not learn, so the step function cannot be trained this way
- the sigmoid slope fades for large , while the ReLU slope stays 1 for every positive
- ReLU has no slope at exactly , but a weighted sum is almost never exactly 0, and is simply taken as 0, so ReLU still trains well
Why do we need this
The activation is what lets a network do more than a weighted sum. The next note shows that without a nonlinear , any number of layers is no better than one.
Next step
Multilayer Perceptron Structure stacks neurons with these activations into layers.
Visual 1: same line, different outputs

How to read it:
- all four panels use the same weighted sum , so the orange line is the same, and darker means a larger output
- step: jumps from 0 to 1 across the line and is flat everywhere else
- piecewise linear and sigmoid: is on the line and changes gradually, linearly inside the ramp or smoothly, as the dashed contours show
- ReLU: is 0 on one whole side and keeps growing on the other, so its scale runs past 1 (contours and )
- so the activation does not move the boundary, it only decides how the output changes away from it, the output range column of Step 2
Visual 2: where each activation can learn

How to read it:
- each panel plots the slope , the factor that multiplies every weight change in Backpropagation
- grey regions have slope 0, so a sample landing there changes no weight and the neuron does not learn from it
- the step function is grey everywhere, which is why it cannot be trained by gradient descent
- the sigmoid slope is never 0 but is below 0.05 for , so a saturated neuron learns very slowly
- ReLU keeps slope 1 for every positive , the slope column of the table in Step 2