GloVe

Global Vectors for Word Representation (GloVe) is an unsupervised machine learning algorithm used for generating word embeddings. It is designed to capture the global co-occurrence statistics of words from a large corpus of text.

• it counts the frequency of each word appearing in the context of every other word in a fixed window size
• the counting pass is done once over the whole corpus, and the vectors are then fitted to those counts

Co-occurrence matrix

A co-occurrence matrix $X$ is constructed, where a cell $X_{ij}$ is a "strength" which represents how often the word $i$ appears in the context of the word $j$.

• a large $X_{ij}$ means the 2 words are seen together often, so their vectors should end up close
• the matrix is built from the entire corpus, so a single pair of words carries the evidence of every window it appeared in

Objective

For each pair of words $i$ and $j$, a cost term is built as follows:

$$w_i^T{w}_j+b_i+b_j-\log (X_{ij})$$

Where $b_i$ and $b_j$ are scalar bias terms associated with words $i$ and $j$ respectively.

To generate these vectors, an objective function $J$ is minimised, which evaluates the sum of all squared errors based on the above equation, weighted with a function $f$:

$$J=\sum _{i,j=1}^{V}f(X_{ij})(w_i^T{\tilde{w}}_j+b_i+{\tilde{b}}_j-\log X_{ij}{)}^2$$

Where $V$ is the size of the vocabulary.

The function $f$ is used to prevent the model from being overly influenced by very common word pairs. A typical form is:

$$f(x)=\{\begin{array}{ll}{\left(\frac{x}{x_{max}}\right)}^{\alpha }& ifx<x_{max}\\ 1& otherwise\end{array}$$

• the dot product of 2 word vectors is being fitted to the logarithm of how often those words co-occur
• pairs above $x_{max}$ are all given the same weight of 1, so no pair can dominate the fit however common it is

Code

import gensim.downloader as api
 
# Load the pre-trained GloVe model (you may need to download it first)
glove_model = api.load("glove-wiki-gigaword-100")
 
# Find the embedding for a specific word
word = "nero"
try:
    embedding = glove_model[word]
    print(f"Embedding for '{word}':")
    print(embedding)
except KeyError:
    print(f"'{word}' is not in the vocabulary.")
 
# Find the most similar words to a given word
similar_words = glove_model.most_similar(word)
print(f"\nWords most similar to '{word}':")
for similar_word, score in similar_words:
    print(similar_word, score)

• gensim.downloader fetches a pre-trained model, so no training is done here
• glove-wiki-gigaword-100 names the corpus and the vector size, where 100 is the number of numbers per word
• a lookup raises KeyError for a word outside the vocabulary, which is why the call sits in a try block
• most_similar returns words ranked by cosine similarity, with the score beside each one

Comparison with Word2Vec

Point	GloVe	Word2Vec
approach	count based matrix factorisation	prediction with a neural network
what it sees	global co-occurrence counts over the whole corpus	1 sliding window at a time
training input	the co-occurrence matrix	the raw text, window by window

In some sense GloVe goes beyond Word2Vec, by not only considering local context but also aggregating the results to a global count.

GloVe factorises a co-occurrence matrix, which is the same kind of count-based decomposition used by Latent Sementic Analysis (LSA).