Global Vectors for Word Representation (GloVe) is an unsupervised machine learning algorithm used for generating word embeddings. It is designed to capture the global co-occurrence statistics of words from a large corpus of text.

  • it counts the frequency of each word appearing in the context of every other word in a fixed window size
  • the counting pass is done once over the whole corpus, and the vectors are then fitted to those counts

Co-occurrence matrix

A co-occurrence matrix is constructed, where a cell is a "strength" which represents how often the word appears in the context of the word .

  • a large means the 2 words are seen together often, so their vectors should end up close
  • the matrix is built from the entire corpus, so a single pair of words carries the evidence of every window it appeared in

Objective

For each pair of words and , a cost term is built as follows:

Where and are scalar bias terms associated with words and respectively. The tilde marks the context version of a word, so is the context vector of word and is the word vector of word .

To generate these vectors, an objective function is minimised, which evaluates the sum of all squared errors based on the above equation, weighted with a function :

Where is the size of the vocabulary.

The function is used to prevent the model from being overly influenced by very common word pairs. A typical form is:

  • the dot product of 2 word vectors is being fitted to the logarithm of how often those words co-occur
  • pairs above are all given the same weight of 1, so no pair can dominate the fit however common it is

Training Example

GloVe makes word vectors in 2 stages. First it counts, then it fits the vectors to the counts. GloVe never goes back to the text after the count stage.

Stage 1: Count once. GloVe reads the full corpus 1 time. It counts how often each word appears near each other word. Here is an example with a window of 1:

  • "the cat sat on the mat"
  • "the dog sat on the rug"
Wordsatonthe
cat101
dog101
sat020

This table is the co-occurrence matrix . "cat" and "dog" have the same row, because they appear near the same words.

Stage 2: Fit the vectors to the counts.

  1. give each word a random vector and a random bias
  2. for each pair of words, calculate the dot product of the 2 vectors and add the 2 biases
  3. compare that value with , and the difference is the error
  4. square the error and multiply it by the weight
  5. adjust the vectors and biases to make the total error smaller
  6. do steps 2 to 5 again many times, over all the pairs

The target for each pair is:

When 2 words appear together often, their dot product must become large. This moves their vectors close together. "cat" and "dog" have the same counts, so they get the same targets, and their vectors become similar.

Why use the log? Counts can be in a range from 1 to millions. The log makes this range much smaller, so the very large counts do not control the fit.

Why use the weight :

  • very common pairs, such as "the" with "of", get a weight limit of 1, so they cannot control the fit
  • pairs with a count of 0 get , so the model skips them, and this is necessary because has no value
  • rare pairs get a small weight, because 1 occurrence can be noise

Code

import gensim.downloader as api
 
# Load the pre-trained GloVe model (you may need to download it first)
glove_model = api.load("glove-wiki-gigaword-100")
 
# Find the embedding for a specific word
word = "nero"
try:
    embedding = glove_model[word]
    print(f"Embedding for '{word}':")
    print(embedding)
except KeyError:
    print(f"'{word}' is not in the vocabulary.")
 
# Find the most similar words to a given word
similar_words = glove_model.most_similar(word)
print(f"\nWords most similar to '{word}':")
for similar_word, score in similar_words:
    print(similar_word, score)
  • gensim.downloader fetches a pre-trained model, so no training is done here
  • glove-wiki-gigaword-100 names the corpus and the vector size, where 100 is the number of numbers per word
  • a lookup raises KeyError for a word outside the vocabulary, which is why the call sits in a try block
  • most_similar returns words ranked by cosine similarity, with the score beside each one

Comparison with Word2Vec

PointGloVeWord2Vec
approachcount based matrix factorisationprediction with a neural network
what it seesglobal co-occurrence counts over the whole corpus1 sliding window at a time
training inputthe co-occurrence matrixthe raw text, window by window
target value1 for a real neighbour, 0 for a negative sample
errorsquared differencesigmoid error

In some sense GloVe goes beyond Word2Vec, by not only considering local context but also aggregating the results to a global count.

GloVe factorises a co-occurrence matrix, which is the same kind of count-based decomposition used by Latent Sementic Analysis (LSA).