Smoothing

Smoothing gives a small amount of probability to events that were never seen in training.

Let's say our training text is 2 sentences:

1. the cat sat
2. the dog sat

Problem

• n-gram probabilities are learned by counting, which is maximum likelihood estimation
• so it counts the dog once and the cat once
• maximum likelihood gives all the mass to observed events, so anything unseen like the bird gets a probability of $0$
• so P(bird|the) = $0/2$ = $0$
• one unseen n-gram makes the probability of the whole sentence $0$, even when the sentence is completely normal and correct

Solution: Smoothing

• it moves a little mass from the seen events to the unseen ones
• meaning we add 1 to every count before dividing
• so the count of the bird goes from $0$ to $1$
• the denominator grows too, so the pairs we really saw did lose a bit of probability
• the bird now gets a small probability instead of $0$
• the model can then score the sentence

The zero probability problem it fixes comes from the counting step in N-gram Language Models.

Add-one smoothing is used outside language models too, because Naive Bayes (NB) applies it to word counts for the same reason.