Stemming

from nltk.stem import PorterStemmer

• porterstemmer is one of the most popular stemming methods proposed by Martin Porter in 1980
• it simplifies words into their root form

stemmer = PorterStemmer()

• pass the PorterStemmer function to a variable stemmer

stemmer.stem(w) for w in words

• we use the stem function by passing w variable used in the for loop

Key Disadvantages

• stemming is just cutting of the text up
• the stem is not necessarily a real word

1. Overstemming, this happens when unrelated words collapse to the same stem. For example: university and universal --> universi. This problem is called precision loss where now the model can't tell them apart.
2. Understemming. related words fail to collapse to the same stem. alumnus, alumni, alumnae get different stems. This problem is called recall loss.
3. Language Challenges. The more complex a language's morphology, the harder the stemmer is to design. French is more confusing so its harder.

Example

import nltk
from nltk.stem import PorterStemmer
 
stemmer = PorterStemmer()
 
words = ["running", "runner", "ran", "easily", "fairness", "studies", "university", "universal", "universe"]
 
stemmed = [stemmer.stem(w) for w in words]
 
print(list(zip(words, stemmed))) # notice the last 3 words become the same after stemming which is the cause of overstemming

[('running', 'run'), ('runner', 'runner'), ('ran', 'ran'), ('easily', 'easili'), ('fairness', 'fair'), ('studies', 'studi'), ('university', 'univers'), ('universal', 'univers'), ('universe', 'univers')]

text = "The studies show that studying and studious students study easily."
tokens = word_tokenize(text)
stemmed_tokens = [stemmer.stem(t) for t in tokens]
 
print(stemmed_tokens)

['the', 'studi', 'show', 'that', 'studi', 'and', 'studiou', 'student', 'studi', 'easili', '.']

words2 = ["alumnus", "alumni", "alumnae", "alumna"]
print([stemmer.stem(w) for w in words2])

['alumnu', 'alumni', 'alumna', 'alumna']

Stemming cuts suffixes with rules alone, and the dictionary-based alternative is Lemmatization.