Skip-gram is 1 of the 2 Word2Vec models. Its goal is to predict context words from a target word, so it runs in the opposite direction to CBOW.
With a window of 2 words before and after, the model takes the target word as its only input and aims to predict each of the 4 surrounding context words.
Structure
- 1 input, which is the target word, and several outputs, which are the context words
- each target and context pair becomes its own training row, so 1 window position produces several rows
- more data is generated using the same sliding window, which is why Skip-gram learns more from the same corpus and takes longer to train
When to use
- Skip-gram is often preferred when you have a large dataset with a rich vocabulary and you want to capture semantic relationships between words effectively
- it can capture rare words and infrequent word associations better than CBOW
- the Skip-gram model is preferred when training on large datasets
A rare word appears in few windows. In CBOW it is 1 input among several and its signal is averaged away, while in Skip-gram it is the sole input of its own rows, so it gets a full update each time.
Comparison with CBOW
| Point | Skip-gram | CBOW |
|---|---|---|
| direction | target to context | context to target |
| inputs and outputs | 1 input, many outputs | many inputs, 1 output |
| rows per window | 1 for each context word | 1 |
| speed | slower | faster |
| rare words | handled better | signal gets averaged away |
In code the choice is made with sg=1 for Skip-gram. The opposite direction is covered in Continuous Bag of Words (CBOW).