Contrastive learning trains a dense retriever like a multiple-choice quiz. The model sees a question with one correct passage and several wrong ones, and it is rewarded for picking the correct one.

Introduction

Contrastive learning learns a representation by contrasting positive pairs (question, correct passage) with negative pairs (question, wrong passage).

  • each training step pulls the question vector towards the correct passage
  • the same step pushes the question vector away from the wrong passages
  • it is the same idea as negative sampling in Word2Vec, where real context words are told apart from random ones

Kinds of negatives

  • in-batch negatives: the correct passages of the other questions in the same training batch serve as free wrong answers
  • hard negatives: wrong passages that look right, for example KYC-E054 for a question about KYC-E052, used in Dense Passage Retrieval (DPR) by Karpukhin et al. in 2020, and they teach fine distinctions

InfoNCE loss

NCE (noise-contrastive estimation) learns by telling real data apart from randomly drawn noise samples. InfoNCE is the multiple-choice version: pick the one real sample among candidates.

PieceMeaning
Similarity of question and passage (a dot product), where is the correct passage and are the wrong ones
the fractionA softmax: the probability that the model picks the correct passage out of the candidates
The loss, which is 0 when the model is certain of the correct passage and large when it prefers a wrong one
(temperature)How strongly gaps between similarity scores are magnified

Remembering tip

Temperature is a magnifying glass on similarity gaps. A small magnifies the gaps, so the same similarities give a much more confident answer.

Worked example

One question, its correct passage and 2 wrong passages, with similarities , and .

  • at the probability of the correct passage is about and the loss is about
  • at the probability is about and the loss is about

The correct passage is the most similar in both cases, yet at the model is only 44% sure of it. With a small , the loss that remains comes mainly from hard negatives that are nearly as similar as the correct passage. If one negative is changed to a hard negative with similarity , the loss at rises to about .

Code

import numpy as np
 
def softmax(scores):
    scores = np.asarray(scores, dtype=float)
    e = np.exp(scores - scores.max())
    return e / e.sum()
 
def info_nce(sim_pos, sim_negs, tau):
    """InfoNCE loss for one question: -log P(correct passage) under a softmax with temperature tau."""
    sims = np.array([sim_pos] + list(sim_negs)) / tau   # divide every similarity by the temperature
    p_correct = softmax(sims)[0]                        # probability given to the correct passage
    return -np.log(p_correct), p_correct
 
for tau in [1.0, 0.1]:
    loss, p = info_nce(0.8, [0.5, 0.2], tau)
    print(f"tau = {tau:<4}  P(correct) = {p:.3f}   loss = {loss:.3f}")
# tau = 1.0   P(correct) = 0.437   loss = 0.828
# tau = 0.1   P(correct) = 0.950   loss = 0.051
 
loss, p = info_nce(0.8, [0.78, 0.2], 0.1)   # one hard negative
print(round(loss, 2))                        # 0.6

Notes:

  • the correct passage is placed at index 0, so softmax(sims)[0] is its probability
  • dividing by tau before the softmax is what applies the temperature

Once the vectors are trained, a collection with millions of passages needs a fast way to search them, see Approximate Nearest Neighbour (ANN) Search.