Byte pair encoding is the subword tokenizer used by GPT. It builds a vocabulary of reusable word pieces instead of whole words, so a word it has never seen can still be written as a few pieces.
How it works
- start from single characters or bytes
- count every adjacent pair in the training corpus
- merge the most frequent adjacent pair into one new symbol
- repeat the merge until the vocabulary reaches the target size
- more vocabulary means we have shorter pairs of bytes
The ordered list of merges is the tokenizer.
A few important Facts
- the rule is always the most frequent adjacent pair, and never the rarest or the longest
- the tokenizer is learned before the language model and then frozen, so it is never trained jointly with the model
See Tokenization Schemes for how subword compares with character, word and byte level.