Bag of Words
- bag of words is a way to turn text into numbers so a machine learning algorithm can work with it
- the idea is to represent a document purely by which words it contains and how many times, throwing away grammar, word order, and sentence strcture entirely
Step by step:
- apply neceesary preprocessing techniques
- build the vocabulary by collecting every unique word in the documents
- represent each document as a vector
Note how this is different compared to the word embedding where each word gets a vector. For BoW, its just 1 vector per document.
This whole portion until the Sklearn part, is just preprocessing of text data, via regex, normalization.
import re
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwordsstop_words = set(stopwords.words('english'))text = "The movie was great. The acting was great too."cleaned = []
for sent in sent_tokenize(text):
sent = re.sub(r'[^a-zA-Z]', ' ', sent) # negation of anything other than a-z or A-Z, replace with space
tokens = [w.lower() for w in word_tokenize(sent) if w.lower() not in stop_words] # tokenize words and remove stop words
cleaned.append(" ".join(tokens)) # join the tokens back into a string and append to cleaned list
print(cleaned)['movie great', 'acting great']
Now we need to use Sklearn to convert the vocabulary into a single vector
from sklearn.feature_extraction.text import CountVectorizervectorizer = CountVectorizer()
X = vectorizer.fit_transform(cleaned)print("Vocabulary:", vectorizer.get_feature_names_out())
print("Matrix:\n", X.toarray())Vocabulary: ['acting' 'great' 'movie']
Matrix:
[[0 1 1]
[1 1 0]]
This is BoW, where each document can be represented as a single matrix, with columns as the total number of unique words, and rows as the total number of sentences, with each value representing the occurances of these words in each sentence.
Cosine Similarity
- this measures how similar 2 vectors are by looking at the angle between them, ignoring their lengths/magnitudes
- For BoW vectors, that is "how similar are the 2 documents" word usage patterns, regardeless of how long each document is
import numpy as npdef cosine_sim(a,b):
return np.dot(a,b) / (np.linalg.norm(a) * np.linalg.norm(b))vecs = X.toarray()
print(cosine_sim(vecs[0], vecs[1]))0.4999999999999999
N-grams - as a probabilistic model, not just string generation
- it is a continuous sequences of n tokens from a text
- the 'n' is just how many tokens we group together at a time
Example:
-
Unigram (n=1): "I", "am", "the", "one", "who", "knocks"
-
Bigram (n=2): "I am", "am the", "the one", "one who", "who knocks"
-
Trigram (n=3): "I am the", "am the one", "the one who", "one who knocks"
-
n-grams also lets us predict the next word based on previous words
-
in otherwords, its a probability model
-
it works by seeing how many times each word appears
-
for example from below, "is" appeared twice, followed by "dog" and "cat" once each. So probability of the word "cat" or "dog" to appear after "is" is 50% each
-
with bigger n, we can capture more context but we would have seen that exact longer sequence before
corpus = ["This is a dog", "This is a cat", "I love my cat"]
tokenized = [['<s>'] + sent.lower().split() + ['</s>'] for sent in corpus]
from collections import Counter
unigram_counts = Counter(w for sent in tokenized for w in sent)
bigram_counts = Counter((sent[i], sent[i+1]) for sent in tokenized for i in range(len(sent)-1))
def bigram_prob(w1, w2):
return bigram_counts[(w1, w2)] / unigram_counts[w1]
print("P(is|This) =", bigram_prob('this', 'is'))
print("P(cat|a) =", bigram_prob('a', 'cat'))
print("P(dog|a) =", bigram_prob('a', 'dog'))P(is|This) = 1.0
P(cat|a) = 0.5
P(dog|a) = 0.5
A count matrix treats every term as equally important, and Term Weighting Schemes correct that by scoring each term. A count matrix is also wide and sparse, and Word Embeddings replace it with a short dense vector for each word.