RoPE is the modern way to tell a Transformer where each token sits in the sequence. Instead of adding a position vector, it rotates the query and key vectors by an angle that depends on the position.
Why it is used
- attention on its own is blind to order, so position must be supplied
- rotating Q and K makes the attention score depend on the distance between two tokens
- many modern decoder language models use RoPE in place of learned absolute positions
Some Pointers
- the original 2017 Transformer used fixed sinusoids
- GPT-1 used learned absolute position embeddings, and not RoPE
- RoPE came later
The sinusoidal scheme it replaced is described in Transformers.