Tokenization

import nltk
nlkt.download('punkt_tab')
from nltk.tokenize import word_tokenize, sent_tokenize

• this stands for the Natural Language Toolkit

Functions

sent_tokenize(string)

• what this does is that we pass it a string containing sentences
• this function will split the string up into a list of sentences
• it will be able to tell that Dr. Tan's . is not a fullstop, and wouldn't consider it a seperate sentence

word_tokenize(string)

• this will tokenize every single word
• fullstops and commas also have a token each.
• Dr. is considered 1 single token

Edge Case: Out of Vocabulary (OOV) handling with <UNK>

UNK_TOKEN = 'UNK' # this represents an unknown token used for unknown or OOV words
vocabulary = {'the', 'cat', 'sat', 'on', 'mat'}
 
tokens = word_tokenize("the cat sat on the rug")
processed = [tok if tok.lower() in vocabulary else UNK_TOKEN for tok in tokens]
print(processed)

['the', 'cat', 'sat', 'on', 'the', 'UNK']

• the subword vocabulary that GPT uses is built by Byte Pair Encoding (BPE)

What counts as 1 token is a design choice, and the options are compared in Tokenization Schemes.