Special tokens are tokens added by the tokenizer that carry an instruction rather than a word of text. This is what control semantics means: each one marks a boundary or a role.
5 Important Types
<BOS>marks the start of a sequence, and an n-gram model prepends n-1 of them to every sentence<EOS>marks the end of a sequence<PAD>fills a short sequence up to the batch length<SEP>separates two segments<UNK>replaces a word that is outside the fixed vocabulary
Importance of EOS
- without an EOS token an n-gram model defines one distribution per sequence length
- with an EOS token it defines one distribution over strings of any length, because the model can decide to stop
The tokenizer that adds these tokens is described in Tokenization.
Start and end markers matter most when sequences are counted, which is what N-gram Language Models do.