Byte pair encoding is the subword tokenizer used by GPT. It builds a vocabulary of reusable word pieces instead of whole words, so a word it has never seen can still be written as a few pieces.
How it works
- start from single characters or bytes
- count every adjacent pair in the training corpus
- merge the most frequent adjacent pair into one new symbol
- repeat the merge until the vocabulary reaches the target size
- more vocabulary means we have shorter pairs of bytes
The ordered list of merges is the tokenizer.
A few important Facts
- the rule is always the most frequent adjacent pair, and never the rarest or the longest
- the tokenizer is learned before the language model and then frozen, so it is never trained jointly with the model
Byte-level BPE and the 256 base values
Byte-level BPE starts from 256 possible byte values. Text is encoded as bytes before the merge rules are applied.
- A Unicode character can require several bytes
- The 256 values form the base byte alphabet; learned merges add more vocabulary entries
- Punctuation, special characters, and emoji can be represented through their bytes
- This complete byte alphabet lets the tokenizer encode text without an unknown-character placeholder
The precise term is 256 byte values. Unicode contains many more characters. Tokenizer documentation
Comparison with WordPiece
| Method | Vocabulary-building rule | Model examples |
|---|---|---|
| BPE | Merge the most frequent adjacent pair | GPT and RoBERTa |
| WordPiece | Choose subwords using a likelihood-based criterion | BERT |
Both methods build reusable subword units. Byte-level BPE also gives complete coverage of the byte alphabet.
See Tokenization Schemes for how subword compares with character, word and byte level.