Contrastive learning trains a dense retriever like a multiple-choice quiz. The model sees a question with one correct passage and several wrong ones, and it is rewarded for picking the correct one.
Introduction
Contrastive learning learns a representation by contrasting positive pairs (question, correct passage) with negative pairs (question, wrong passage).
- each training step pulls the question vector towards the correct passage
- the same step pushes the question vector away from the wrong passages
- it is the same idea as negative sampling in Word2Vec, where real context words are told apart from random ones
Kinds of negatives
- in-batch negatives: the correct passages of the other questions in the same training batch serve as free wrong answers
- hard negatives: wrong passages that look right, for example KYC-E054 for a question about KYC-E052, used in Dense Passage Retrieval (DPR) by Karpukhin et al. in 2020, and they teach fine distinctions
InfoNCE loss
NCE (noise-contrastive estimation) learns by telling real data apart from randomly drawn noise samples. InfoNCE is the multiple-choice version: pick the one real sample among candidates.
| Piece | Meaning |
|---|---|
| Similarity of question and passage (a dot product), where is the correct passage and are the wrong ones | |
| the fraction | A softmax: the probability that the model picks the correct passage out of the candidates |
| The loss, which is 0 when the model is certain of the correct passage and large when it prefers a wrong one | |
| (temperature) | How strongly gaps between similarity scores are magnified |
Remembering tip
Temperature is a magnifying glass on similarity gaps. A small magnifies the gaps, so the same similarities give a much more confident answer.
Worked example
One question, its correct passage and 2 wrong passages, with similarities , and .
- at the probability of the correct passage is about and the loss is about
- at the probability is about and the loss is about
The correct passage is the most similar in both cases, yet at the model is only 44% sure of it. With a small , the loss that remains comes mainly from hard negatives that are nearly as similar as the correct passage. If one negative is changed to a hard negative with similarity , the loss at rises to about .
Code
import numpy as np
def softmax(scores):
scores = np.asarray(scores, dtype=float)
e = np.exp(scores - scores.max())
return e / e.sum()
def info_nce(sim_pos, sim_negs, tau):
"""InfoNCE loss for one question: -log P(correct passage) under a softmax with temperature tau."""
sims = np.array([sim_pos] + list(sim_negs)) / tau # divide every similarity by the temperature
p_correct = softmax(sims)[0] # probability given to the correct passage
return -np.log(p_correct), p_correct
for tau in [1.0, 0.1]:
loss, p = info_nce(0.8, [0.5, 0.2], tau)
print(f"tau = {tau:<4} P(correct) = {p:.3f} loss = {loss:.3f}")
# tau = 1.0 P(correct) = 0.437 loss = 0.828
# tau = 0.1 P(correct) = 0.950 loss = 0.051
loss, p = info_nce(0.8, [0.78, 0.2], 0.1) # one hard negative
print(round(loss, 2)) # 0.6Notes:
- the correct passage is placed at index 0, so
softmax(sims)[0]is its probability - dividing by
taubefore the softmax is what applies the temperature
Once the vectors are trained, a collection with millions of passages needs a fast way to search them, see Approximate Nearest Neighbour (ANN) Search.