Introduction

The activation function turns a neuron's weighted sum into its output. Its shape decides what the network can represent and how easily it can be trained. This note covers the four common choices.

Main idea

The four common activations are the binary step, piecewise linear, sigmoid $f(v)=\frac{1}{1+\exp(-av)}$ and ReLU $f(v)=\max(0,v)$. All four are nonlinear. The step gives a hard yes or no, the sigmoid is a smooth version of it, and ReLU passes positive values and blocks negative ones.


Overview

Every activation maps one number to one number, applied to each neuron separately:

The four choices differ in two things: the range of the output, and whether the function is smooth (has a slope everywhere that training can use).


Step 1: four activation functions

Binary (step):

  • output 1 when , else 0
  • a hard yes or no, which is the neuron as a pure linear classifier
  • flat everywhere except the jump, so its slope is zero almost everywhere

Piecewise linear:

  • 0 below a lower limit, 1 above an upper limit, and a straight ramp in between (the limits in the figure are only an example)
  • a softened step, linear only inside the ramp

Sigmoid:

  • a smooth S-curve from 0 to 1, passing at
  • sets the steepness: a larger makes it closer to the step
  • for large positive , so , and for large negative , so
  • it has a slope everywhere, which gradient-based training needs

ReLU (rectified linear unit):

  • passes positive values unchanged and sets negative values to 0
  • the output has no upper limit
  • the simplest nonlinear function, and very cheap to compute
  • it is linear on each side of 0, but the corner at 0 makes it nonlinear overall, since in general

Step 2: comparing them

FunctionOutput rangeSmoothSlope
binaryno0 except at the jump
piecewise linearno, corners at the limitsconstant on the ramp, 0 outside
sigmoidyeslargest at , near 0 for large
ReLUno, one corner at 01 for , 0 for
linear, all real numbersyes1 everywhere

The linear function is not one of the four. It is only used at the output, when the network must produce any real number, because a hidden layer with it collapses (Multilayer Perceptron Structure).

What the slope means for training:

  • training changes the weights in proportion to the slope , shown in Backpropagation
  • where the slope is 0, the neuron does not learn, so the step function cannot be trained this way
  • the sigmoid slope fades for large , while the ReLU slope stays 1 for every positive
  • ReLU has no slope at exactly , but a weighted sum is almost never exactly 0, and is simply taken as 0, so ReLU still trains well

Why do we need this

The activation is what lets a network do more than a weighted sum. The next note shows that without a nonlinear , any number of layers is no better than one.


Next step

Multilayer Perceptron Structure stacks neurons with these activations into layers.


Visual 1: same line, different outputs

How to read it:

  • all four panels use the same weighted sum , so the orange line is the same, and darker means a larger output
  • step: jumps from 0 to 1 across the line and is flat everywhere else
  • piecewise linear and sigmoid: is on the line and changes gradually, linearly inside the ramp or smoothly, as the dashed contours show
  • ReLU: is 0 on one whole side and keeps growing on the other, so its scale runs past 1 (contours and )
  • so the activation does not move the boundary, it only decides how the output changes away from it, the output range column of Step 2

Visual 2: where each activation can learn

How to read it:

  • each panel plots the slope , the factor that multiplies every weight change in Backpropagation
  • grey regions have slope 0, so a sample landing there changes no weight and the neuron does not learn from it
  • the step function is grey everywhere, which is why it cannot be trained by gradient descent
  • the sigmoid slope is never 0 but is below 0.05 for , so a saturated neuron learns very slowly
  • ReLU keeps slope 1 for every positive , the slope column of the table in Step 2