Normalization

Normalization is a broad term for standardizing words in a string. A few pointers we can work on are:

1. casing (upper or lower case)
2. expanding contractions (like you're to you are)
3. Stemming
4. Lemmatization

1. Casing

simply use upper() and lower()

2. Expanding Contractions

from contractions import fix
 
text = "I'd still recommend it, even though it isn't perfect and they're a bit slow."
expanded = fix(text)
print(expanded) # this just allows us to expand the constractions like I'd, isn't, etc.

I would still recommend it, even though it is not perfect and they are a bit slow.

3. Checking if order of expansion and lowercase matters

raw = "It isn't great, but it's not bad either."
 
a = fix(raw.lower())
b = fix(raw).lower()
 
print(a)
print(b)

it is not great, but it is not bad either.
it is not great, but it is not bad either.

4. Link to stopwords and negation

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))
tokens = word_tokenize(fix(raw).lower())
filtered = [w for w in tokens if w not in stop_words]
 
print(filtered)

['great', ',', 'bad', 'either', '.']