Transformer-XL

Transformer-XL is a Transformer model designed to learn dependencies across long text sequences. It carries information from one text segment to the next.

Segment-level recurrence

1. Process one segment of text
2. Cache its hidden states
3. Process the next segment with access to the cached states
4. Reuse this process across later segments

segment 1 -> hidden states -> cached memory
                                   |
segment 2 -------------------------+-> next hidden states

The memory contains representations of earlier text. Attention can use these representations when it processes a later segment.

Relative positional encoding

The original Transformer uses absolute positional encodings. Transformer-XL uses relative positional information.

• Absolute position describes where a token occurs in the sequence
• Relative position describes the distance between token positions
• Relative positions help attention use cached states from earlier segments

Connection to XLNet

XLNet uses Transformer-XL's segment recurrence and relative positional encoding. XLNet adds a permutation language modelling objective.