Lemmatization

• reducing words to their DICTIONARY FORM (lemma)
• using the word's Part of Speech (POS) as context
• Lemmatization is more accurate than Stemming but require POS info to work properly
• POS is a category that describes the role that a particular word plays in a sentence
• However, the default POS tagger assumes all words in the string are nouns

POS Tagging is the action of assigning (tagging) a grammatical label to a particular word in a sentence.

nltk.stem import WordNetLemmatizer

• this shows how we import it

lemmatizer = WordNetLemmatizer()

• callback function passed to variable lemmatizer

lemmatizer.lemmatize(w) for w in words

• this is how we use the lemmatizer function

pos="v"

• means that we force all words to be a verb
• POS tells the lemmatizer which dictionary to look the word up in
• WordNet stores lemmas seperately per part of speech, so the same surface word has different lemmas depending on its grammatical role

pos_tag()

• This function assigns a POS tag for each token

The Problem with POS Tagging

• words have different meaning when used differently in different sentence structure
• Meaning the same word can have different POS tags depending on how its used and where its used in

After knowing this last normalization technique, we now can apply all these techniques and perform Vectorization.

Example

import nltk
from nltk.stem import WordNetLemmatizer
from nltk.stem import PorterStemmer
 
nltk.download('wordnet')
 
lemmatizer = WordNetLemmatizer()
words = ["running", "runner", "ran", "easily", "fairness", "studies", "university", "universal", "universe"]
 
lemmatized = [lemmatizer.lemmatize(w) for w in words]
 
print(list(zip(words, lemmatized)))

[('running', 'running'), ('runner', 'runner'), ('ran', 'ran'), ('easily', 'easily'), ('fairness', 'fairness'), ('studies', 'study'), ('university', 'university'), ('universal', 'universal'), ('universe', 'universe')]

print(lemmatizer.lemmatize("running", pos="v"))
print(lemmatizer.lemmatize("ran", pos="v"))
print(lemmatizer.lemmatize("better", pos="a"))

run
run
good