Vectorization

Bag of Words

bag of words is a way to turn text into numbers so a machine learning algorithm can work with it
the idea is to represent a document purely by which words it contains and how many times, throwing away grammar, word order, and sentence strcture entirely

Step by step:

apply neceesary preprocessing techniques
build the vocabulary by collecting every unique word in the documents
represent each document as a vector

Note how this is different compared to the word embedding where each word gets a vector. For BoW, its just 1 vector per document.

This whole portion until the Sklearn part, is just preoprocessing of text data, via regex, normalization.

import re
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))

text = "The movie was great. The acting was great too."

cleaned = []
for sent in sent_tokenize(text):
    sent = re.sub(r'[^a-zA-Z]', ' ', sent) # negation of anything other than a-z or A-Z, replace with space
    tokens = [w.lower() for w in word_tokenize(sent) if w.lower() not in stop_words] # tokenize words and remove stop words
    cleaned.append(" ".join(tokens)) # join the tokens back into a string and append to cleaned list
print(cleaned)

['movie great', 'acting great']

Now we need to use Sklearn to convert the vocabulary into a single vector

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(cleaned)

print("Vocabulary:", vectorizer.get_feature_names_out())
print("Matrix:\n", X.toarray())

Vocabulary: ['acting' 'great' 'movie']
Matrix:
 [[0 1 1]
 [1 1 0]]

Cosine Similarity

this measures how similar 2 vectors are by looking at the angle between them, ignoring their lengths/magnitudes
For BoW vectors, that is "how similar are the 2 documents" word usage patterns, regardeless of how long each document is

import numpy as np

def cosine_sim(a,b):
    return np.dot(a,b) / (np.linalg.norm(a) * np.linalg.norm(b))

vecs = X.toarray()
print(cosine_sim(vecs[0], vecs[1]))

0.4999999999999999

N-grams - as a probabilistic model, not just string generation

it is a continuous sequences of n tokens from a text
the 'n' is just how many tokens we group together at a time

Example:

Unigram (n=1): "I", "am", "the", "one", "who", "knocks"
Bigram (n=2): "I am", "am the", "the one", "one who", "who knocks"
Trigram (n=3): "I am the", "am the one", "the one who", "one who knocks"
n-grams also lets us predict the next word based on previous words
in otherwords, its a probability model
it works by seeing how many times each word appears
for example from below, "is" appeared twice, followed by "dog" and "cat" once each. So probability of the word "cat" or "dog" to appear after "is" is 50% each
with bigger n, we can capture more context but we would have seen that exact longer sequence before

corpus = ["This is a dog", "This is a cat", "I love my cat"]
tokenized = [['<s>'] + sent.lower().split() + ['</s>'] for sent in corpus]
 
from collections import Counter
unigram_counts = Counter(w for sent in tokenized for w in sent)
bigram_counts = Counter((sent[i], sent[i+1]) for sent in tokenized for i in range(len(sent)-1))
 
def bigram_prob(w1, w2):
    return bigram_counts[(w1, w2)] / unigram_counts[w1]
 
print("P(is|This) =", bigram_prob('this', 'is'))
print("P(cat|a) =", bigram_prob('a', 'cat'))
print("P(dog|a) =", bigram_prob('a', 'dog'))

P(is|This) = 1.0
P(cat|a) = 0.5
P(dog|a) = 0.5

A count matrix treats every term as equally important, and Term Weighting Schemes correct that by scoring each term. A count matrix is also wide and sparse, and Word Embeddings replace it with a short dense vector for each word.

JJ

Explorer