Next Sentence Prediction

Next Sentence Prediction (NSP) is a binary classification task used in BERT pre-training. It predicts whether sentence B follows sentence A in the source text.

Constructing the training pairs

Pair type	Share	Target
Sentence B is the actual next sentence	50 percent	IsNext
Sentence B is sampled randomly	50 percent	NotNext

The task uses sentence pairs taken from unlabelled text. The way the pair is constructed provides the training label.

Input and output

[CLS] Sentence A [SEP] Sentence B [SEP]
       -> BERT encoder
       -> final [CLS] representation
       -> IsNext / NotNext classifier

The output is one class for the sentence pair. The model learns relationships between sentences.

NSP and MLM

• NSP predicts a relationship between two sentences
• Masked Language Modelling predicts selected tokens inside the input
• BERT uses both objectives during pre-training
• RoBERTa removes NSP and retains masked token prediction