Entropy

Motivation

Entropy gives us the uncertainty of a probability distribution.

It connects the model's output distribution and the loss we train it with, so it sits under cross-entropy loss and under Perplexity (PPL).

Surprise of a single outcome

Self-information, or surprise, measures how unexpected one outcome is. We can imagine it like having an inverse relationship with the probability. Imagine we have 9/10 chickens in a farm that are blue in colour. If we randomly picked up a blue chicken, we wouldn't be suprised. Below shows the formula:

I (x) = - lo g p (x)

a token the model gave $p = 0.99$ carries almost no surprise, because $- lo g (0.99) \approx 0.01$
a token the model gave $p = 0.01$ carries a large surprise of $- lo g (0.01) \approx 4.6$
the log is what makes surprise add up across independent events instead of multiply
natural log gives nats, log base 2 gives bits, and you convert by dividing nats by $ln 2$

Entropy is the average surprise

Entropy is the expected surprise over the whole distribution:

H (p) = - x \sum p (x) lo g p (x)

Read it as the average number of yes or no questions you need to identify the outcome.

Behaviour at the two extremes:

a uniform distribution has the highest possible entropy, because every outcome is equally unexpected
a one-hot distribution has zero entropy, because the outcome is already known
a distribution over a vocabulary of size $V$ has a maximum entropy of $lo g V$ ; cross-entropy loss can be larger when the model assigns very low probability to the correct token

Worked feel for the numbers:

Distribution over 3 tokens	Entropy in nats	Interpretation
1/3, 1/3, 1/3	1.10	full uncertainty, 3 effective choices
0.8, 0.1, 0.1	0.64	mostly decided
1.0, 0.0, 0.0	0	no uncertainty left

Cross-entropy is the training loss

Cross-entropy measures the average surprise when the data comes from the true distribution $p$ but the coding scheme comes from the model $q$ :

H (p, q) = - x \sum p (x) lo g q (x)

in language modelling the true distribution is one-hot on the observed next token, so the sum collapses to $- lo g q (correct token)$
that single term is exactly the per-token loss reported by F.cross_entropy
cross-entropy is always at least the true entropy, and the gap is the KL divergence, which is why KL is also called relative entropy
expected cross-entropy over the true distribution is bounded below by its entropy; loss measured on a finite training sample can differ from that expected value

Why this matters in practice

Three practical uses:

the loss curve is a cross-entropy in nats per token, and exponentiating it gives perplexity, an effective branching factor
a high entropy output distribution means the model is undecided, and a low entropy one means it is committed
temperature in decoding is a direct entropy control, since raising $τ$ flattens the distribution and raises its entropy while lowering $τ$ sharpens it

Code

Compare uncertainty with error on one target

import math
 
probabilities = [0.8, 0.1, 0.1]
entropy = -sum(p * math.log(p) for p in probabilities if p > 0)
loss_if_class_0 = -math.log(probabilities[0])
loss_if_class_1 = -math.log(probabilities[1])
print(round(entropy, 3))          # 0.639
print(round(loss_if_class_0, 3))  # 0.223
print(round(loss_if_class_1, 3))  # 2.303

Read the difference:

entropy averages surprise using every class probability in this distribution
loss_if_class_0 uses the probability of the observed target class 0
loss_if_class_1 is larger because the model gives class 1 only probability 0.1
math.log uses the natural logarithm, so the values are in nats
if p > 0 implements the zero contribution of a zero-probability term in entropy

The corresponding framework loss

import torch
logits = torch.log(torch.tensor([[0.8, 0.1, 0.1]]))
loss = torch.nn.CrossEntropyLoss()(logits, torch.tensor([0]))
print(round(loss.item(), 3))  # 0.223

import tensorflow as tf
logits = tf.math.log(tf.constant([[0.8, 0.1, 0.1]]))
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
loss = loss_fn(tf.constant([0]), logits)
print(round(float(loss), 3))  # 0.223

Taking the log of the chosen probabilities supplies logits whose softmax returns that distribution. Both losses select class 0 as the target. PyTorch receives predictions first; Keras receives targets first.

JJ

Explorer