Introduction

Softmax turns a list of scores into positive weights that add up to 1. It looks like a trick of neural networks, but it is the Bayes posterior in disguise: when the scores are log-probabilities, softmax gives exactly the posterior probabilities of the classes. The same normalized weights appear in the Parzen window, the kNN vote, the MLP outputs and attention (the posterior itself is in Prior, Likelihood and Posterior).

Main idea

$\text{softmax}(s)_i=\frac{e^{s_i}}{\sum_j e^{s_j}}$ makes the scores positive, keeps their order and normalizes them to add up to 1. With $s_i=\ln p(\mathbf{x}|\omega_i)+\ln p(\omega_i)$ it is exactly the posterior $p(\omega_i|\mathbf{x})$, so picking the largest softmax output is the MAP rule. Scaling the scores changes how sharp the weights are: large differences put almost all the weight on the winner, small ones spread it evenly.


What it is

Properties:

  • every output is positive and they add up to 1, so they can be read as probabilities
  • the largest score gets the largest weight, so the order is kept
  • adding the same constant to every score changes nothing, only differences between scores matter

Physical meaning

Why it is the posterior:

  • Bayes:
  • write each top as with , the log form used by the discriminant functions
  • then the posterior is exactly

Sharpness:

  • dividing every score by a number before the softmax controls the sharpness
  • small makes the differences large, so almost all the weight goes to the winner, which is the hard MAP decision
  • large makes the differences small, so the weights become nearly equal, a plain average

Where it is used

Note, stepWhat the normalized weights mean there
Prior, Likelihood and Posteriorthe posterior is likelihood times prior, normalized over the classes
Discriminant Functionsthe scores are the log tops of the posterior, so the largest is the largest posterior
Parzen Windowwith a Gaussian kernel, each sample's weight falls with distance, and normalized these weights are a softmax of negative squared distances
kNN Classifierthe vote shares behave like posterior estimates, a counting version of the same normalization
Traditional Neural Network MLPwith 1-of- targets and squared error, the outputs approach the posteriors, so the largest output is the MAP class
Encoder and Decoder Stacksthe last linear layer and softmax give a probability for every word in the vocabulary
Vision Transformer and CNNthe classification head outputs a probability per class
Attention as Weighted Sum of Values, Step 3each row of scores becomes weights that add up to 1, so each output is a weighted average of the value rows; dividing by sets the sharpness

Common traps

Points that are easy to get wrong:

  • softmax outputs are only true posteriors if the scores are true log-probabilities; a network's outputs are estimates
  • attention weights add up to 1 but are not class probabilities: they say how much of each token to mix in
  • the MAP decision only needs the largest score, so the softmax does not change which class wins

Test yourself


Why do we need this

Softmax is how probability enters the neural network half of Part 2. Seeing it as the normalized Bayes posterior connects the first classifier to the last layer of every network, and explains what the attention weights are.


Visual 1: scores to weights, and what scaling does

The same four scores go through the softmax as they are, multiplied by 4, and divided by 4:

  • the scores can be negative and have no fixed total, but every output is positive and each set adds up to 1
  • the order never changes: always gets the largest weight
  • scores make the differences large, so takes of the weight, close to the hard MAP choice
  • scores make them small, so the weights are close to a plain average; dividing attention scores by is this kind of scaling, and it stops them being far too sharp

Visual 2: softmax of log scores is the posterior

Three 1-D Gaussian classes with priors , and , read at one :

  • top: with , each curve is the joint density, the top of Bayes' rule; their sum (dashed) is
  • step 1: the three tops at the chosen ; step 2: the softmax bottom , the three stacked, which is exactly
  • step 3: each top divided by the bottom is , and the three add up to 1
  • the class with the largest top also has the largest posterior, so the MAP decision does not need the division at all