WordPiece is a subword tokenization method used by BERT. It builds a vocabulary of reusable parts of words.
Learning the vocabulary
- Start with small character units
- Consider joining adjacent units into longer subwords
- Prefer additions that improve the likelihood of the training data under the vocabulary
- Continue until the vocabulary reaches its chosen size
BPE chooses the most frequent adjacent pair. WordPiece uses a likelihood-based criterion.
Splitting a new word
With a learned WordPiece vocabulary, select the longest matching piece from the start of the remaining word. Repeat until the word is covered.
BERT uses ## to mark a continuation piece inside a word. The learned vocabulary determines the actual split. WordPiece tokenization
Why subwords help
Related words can share pieces. For example:
tiresome -> tire + some
tired -> tire + dThese splits show the subword idea. A specific tokenizer can produce different splits.
Subwords reduce the number of unknown words. If WordPiece cannot cover a word with its vocabulary, it can return [UNK]. See Tokenization Schemes.