JJ

Word Embeddings

4 min read

To process textual data, we must convert raw text data into meaningful numerical representations. Vectorization methods such as bag of words and TF-IDF already do this, and they carry 2 important drawbacks:

  • they have limited ability in capturing the semantic meaning of the words
  • they are computationally inefficient and require a high-dimensional representation when the corpus is large

A word embedding fixes both by giving each word a short dense vector, where the position of that vector carries meaning.

Embeddings learned within a model

A trainable embedding layer stores a table with one row per vocabulary ID. Training changes these row values through the task loss. The number of vocabulary rows and the number of values per row are separate settings.

An ordered sequence of token IDs preserves token positions. A bag-of-words row records counts across vocabulary columns and loses that order. Passing counts into an embedding lookup would treat each count as a token ID.

Here are the 2 advanced embedding methods that significantly improve on these issues:

  1. Word2Vec
    • a neural network approach to train word embeddings
  2. GloVe
    • a model using a global count-based matrix factorisation approach

Code

PyTorch: IDs select embedding rows

This code shows a general embedding layer. It creates a vector table and retrieves vectors using token IDs.

import torch
from torch import nn
 
corpus = ["I like", "like this movie"]
tokens = [sentence.split() for sentence in corpus]
word_to_id = {
	"I": 1,
	"like": 2,
	"this": 3,
	"movie": 4
}
ids = [[word_to_id[word] for word in sentence] for sentence in tokens]
max_length = max(len(sentence) for sentence in ids)
 
padded_ids = [
    sentence + [0] * (max_length - len(sentence))
    for sentence in ids
]
# [[1, 2, 0], [2, 3, 4]]
 
ids = torch.tensor(padded_ids, dtype=torch.long)
# tensor([[1, 2, 0],
#         [2, 3, 4]])
 
embedding = nn.Embedding(num_embeddings=5, embedding_dim=3, padding_idx=0)
# just makes it into a table:
# |--ID--|---Meaning---|---Vector---|
# embedding dimensions = 3 gives us e.g. [0.2, -0.4, 0.7] for "like"
# pytorch just fills the vectors with random numbers first
 
vectors = embedding(ids) # replaces each ID with its vector
# so from the above ids,
# [1, 2, 0] → [[0.2, -0.4, 0.7],
#              [0.8,  0.1, 0.3],
#              [0.0,  0.0, 0.0]]
 
print(tuple(embedding.weight.shape))  # (5, 3)
print(tuple(vectors.shape))           # (2, 3, 3)
# 2 represents the 2 sentence (rows)
# each of these sentences (rows), contain 3 by 3 matrix
# 3 vector dimensions (rows) with 3 words (columns)
 
print(vectors[0, 2].tolist())         # [0.0, 0.0, 0.0]

From the above code:

  • each unique word has an ID
  • ID 0 is always reserved for <padding>
  • There are five possible IDs (in a sentence with five unique words), from 0 through 4
  • nn.Embedding() creates a table that stores one embedding_dim dimensional vector for each token ID
  • Each row of embedding.weight contains three values
  • Input ID 2 selects row 2, so repeated IDs retrieve the same row
  • (2, 3) means two sequences of three positions; the lookup adds the three-value embedding dimension
  • padding_idx=0 keeps this newly initialised padding row at zero during ordinary gradient updates

PyTorch's padding row setting does not automatically tell a recurrent layer to skip that timestep. Use suitable lengths, packing, or masks when processing padded sequences.

TensorFlow/Keras: a padding mask

import tensorflow as tf
 
# using same sentence and tokenization as before
 
ids = tf.constant([[1, 2, 0], [2, 3, 4]])
# converts nested list into a tensor
 
embedding = tf.keras.layers.Embedding(input_dim=5, output_dim=3, mask_zero=True)
# Creates the layer object that stores these settings:
# 5 possible IDs, 3 values per vector, and ID 0 treated as padding.
 
vectors = embedding(ids)
# creates and randomly initialises the 5 x 3 table
# then retrieves a vector for each input ID, preserving its position
 
mask = embedding.compute_mask(ids)
print(tuple(vectors.shape))  # (2, 3, 3)
print(mask.numpy())
# [[ True  True False]
#  [ True  True  True]]

mask_zero=True marks ID zero as padding for compatible later layers. It creates a mask; the embedding vector at ID zero is not required to contain zeros. Keras Embedding reference.

The trainable table is included in a larger model's learned parameters. The input IDs must identify tokens; count values from a bag-of-words matrix have a different meaning.

Graph View

Backlinks