Tokenization Schemes

A tokenization scheme decides what one token is. The choice sets the vocabulary size and the sequence length, and those two are inversely related.

4 Common Schemes

For example, the word unbelievable.

1. Character level

• one token per character, so u, n, b, e, l, ... which is 12 tokens
• the vocabulary is tiny, about 100 symbols, since that is all the characters there are
• every sequence becomes very long since many tokens per sentence

2. Word level

• one token per word, so unbelievable is 1 token
• the vocabulary is huge, maybe 50,000 entries, and it is still never enough
• this is the open vocabulary problem, because a word you did not see in training, like unbelievableness, has no token and becomes <UNK>

3. Subword level

• one token per common piece, so un + believ + able, which is 3 tokens
• the vocabulary is medium, maybe 32,000 pieces
• a word never seen before can often be represented with known pieces, which reduces unknown words
• BPE and WordPiece both build subword vocabularies
• GPT uses BPE; BERT uses WordPiece
• complete byte coverage in byte-level BPE handles unknown characters; WordPiece can still produce [UNK]

4. Byte level

• one token per byte of the UTF-8 encoding
• the vocabulary is exactly 256, and it covers every language and emoji, so it is universal
• sequences get even longer than character level for non-English text

Shared subwords: tiresome and tired

tiresome -> tire + some
tired    -> tire + d

The shared piece tire lets related word forms reuse part of their representation. Subwords can capture roots, prefixes, and suffixes.

These splits illustrate the idea. The learned vocabulary determines the actual output of a tokenizer.

Method	How it chooses vocabulary additions
BPE	Most frequent adjacent pair
WordPiece	Likelihood-based criterion

Trade Off

• a smaller vocabulary gives longer sequences and a smaller embedding table
• longer sequences also raise attention cost, because attention grows with the square of the sequence length
• a larger vocabulary gives shorter sequences and a larger embedding table

The practical code side of splitting text is in Tokenization.

A word the scheme cannot cover is replaced by a placeholder, and those placeholders are described in Special Tokens.