Word2Vec

Word2Vec's key idea is that words that have similar meanings or are used in similar contexts should have similar vector representations. This enables Word2Vec to capture the semantic relationships between words, so it can represent that "king" is to "queen" as "man" is to "woman".

The 2 models

Word2Vec includes 2 main models, and they are the same idea run in opposite directions.

1. Continuous Bag of Words (CBOW)
   • aims to predict a target word based on its context words
2. Skip-gram
   • predicts context words given a target word

Structure

The Word2Vec model is based on a shallow neural network consisting of an input layer, a densely connected hidden layer and an output layer.

• the hidden layer is also called the projection layer
• the embedding is not the output of the network, it is the weights held in the hidden layer
• the prediction task exists only to force those weights into a useful arrangement

Training

Word2Vec is trained iteratively as follows:

• given a large text corpus, go over the text with a sliding window, moving 1 word at a time
• at each step there is a target word and surrounding context words
• in the Skip-gram setup, compute prediction probabilities of context words based on the target word
• these predictions are updated through standard neural network iterations

The steps inside 1 iteration:

1. create 2 matrices, the Embedding matrix and the Context matrix, and initialise them with random numbers
2. in each training step, take 1 positive example and its associated negative examples
3. find the input words in the embedding matrix, and find the corresponding output words in the context matrix
4. calculate the similarity between input and output words using the dot product
5. pass it through a sigmoid function to convert it into probabilities
6. calculate the loss, where error = target - sigmoid_scores
7. use the error score to adjust the embeddings of the output words, so that the next time this calculation is made the result would be closer to the target scores
8. iteratively perform this through every input word

After training, the context embeddings are discarded, leaving us with the final embedding vector.

Objective

The objective is to maximise the average log-probability of the context words occurring around the input word over the entire vocabulary.

$$\frac{1}{T}\sum _{t=1}^T\sum _{-c\le j\le c,j\ne 0}\log p(w_{t+j}\mid w_t)$$

Where $T$ is all the words in the training data and $c$ is the training context window.

• the inner sum walks the context positions around 1 target word, and $j\ne 0$ skips the target itself
• the outer sum repeats this for every position in the corpus

SoftMax

One way to calculate the above probability is to use the SoftMax function.

$$p(w_O\mid w_I)=\frac{\exp (v_{w_O}^{\prime T}{v}_{w_I})}{{\sum }_{w=1}^W\exp (v_w^{\prime T}{v}_{w_I})}$$

Where $v_w$ and $v_w^{\prime }$ are the vector representations of the word $w$ as the input and output respectively, and $W$ is the number of words in the entire vocabulary.

• the intuition is that words that appear in the same context will have similar vector representations
• the numerator shows this by assigning a larger value for similar words through the dot product of the 2 vectors
• the denominator is a normalising factor that has to be computed over the entire vocabulary, which is extremely difficult to compute for large vocabularies

Negative sampling

Negative sampling is a workaround that aims at maximising the similarity of the words in the same context and minimising it when they occur in different contexts.

• instead of doing the minimisation for all the words in the dictionary except for the context words, it randomly selects a handful of words ($2\le k\le 20$) depending on the training size and uses them to optimise the objective
• a larger $k$ is chosen for smaller datasets and vice versa

$$\log \sigma (v_{w_O}^{\prime T}{v}_{w_I})+\sum _{i=1}^k{E}_{w_i\sim P_n(w)}\left[\log \sigma (-v_{w_i}^{\prime T}{v}_{w_I})\right]$$

Where $\sigma$ is the sigmoid function and $P_n(w)$ is the noise distribution with the negative samples drawn from it. It is calculated as the unigram distribution of the words to the power of 3/4.

$$P_n(w)=\frac{U(w{)}^{\frac{3}{4}}}{Z}$$

Where $Z$ is a normalisation constant.

• the first term pushes the real pair together, and the second term pushes the sampled fake pairs apart
• the power of 3/4 flattens the distribution, so very common words are sampled less often than their raw frequency suggests

Code

from gensim.models import Word2Vec
from nltk.tokenize import word_tokenize
 
sentences = [
    "Word2Vec is a technique for word embedding.",
    "Embedding words in vector space is powerful for NLP.",
    "Gensim provides an easy way to work with Word2Vec.",
]
 
tokenized_sentences = [word_tokenize(sentence.lower()) for sentence in sentences]
 
model = Word2Vec(tokenized_sentences, vector_size=100, window=5, min_count=1, sg=0)
model.save("word2vec.model")

word = "word"
if word in model.wv:
    embedding = model.wv[word]
    print(f"Embedding for '{word}': {embedding}")
else:
    print(f"'{word}' is not in the vocabulary.")
 
similarity = model.wv.similarity("word", "embedding")
print(f"Similarity between 'word' and 'embedding': {similarity}")

Argument	Meaning
vector_size	the length of each word vector, so 100 numbers per word
window	the sliding window size, which sets how many context words count
min_count	the minimum number of occurrences before a word enters the vocabulary
sg	the model choice, where 0 is CBOW and 1 is Skip-gram

• the input is a list of tokenized sentences, so tokenization happens before training
• model.wv holds the trained vectors, and it behaves like a dictionary keyed by the word
• a word outside the vocabulary has no vector at all, so membership must be checked before lookup
• model.wv.similarity returns the cosine similarity between 2 word vectors