Continuous Bag of Words (CBOW) is 1 of the 2 Word2Vec models. Its goal is to predict a target word based on the context words surrounding the target word.
The window sets how much context is used. Looking at the 2 words before and after a target word gives 4 context words, and those 4 words are the inputs while the missing target word is the output.
Reading a sentence with 1 word blanked out is the same task. The surrounding words are enough to guess what belongs in the gap, and the model learns word meaning by being forced to make that guess again and again.
Structure
- several inputs, which are the context words, and 1 output, which is the target word
- the context word vectors are summed at the projection layer before the prediction is made
- summing throws away the order of the context words, which is why the name contains "bag of words"
- each sliding window position produces exactly 1 training row
When to use
- CBOW is computationally more efficient and often trains faster than Skip-gram
- it is a good choice for smaller datasets, or when you want to quickly generate word embeddings
- it can be more effective when the context window size is relatively small, as it directly predicts the target word based on nearby words
Comparison with Skip-gram
| Point | CBOW | Skip-gram |
|---|---|---|
| direction | context to target | target to context |
| inputs and outputs | many inputs, 1 output | 1 input, many outputs |
| rows per window | 1 | 1 for each context word |
| speed | faster | slower |
| best suited to | smaller datasets | large datasets with rich vocabulary |
In code the choice is made with sg=0 for CBOW. The opposite direction is covered in Skip-gram.