Transformer-XL is a Transformer model designed to learn dependencies across long text sequences. It carries information from one text segment to the next.

The main problem is that vanilla transformers process a fixed number of tokens, for example 512 tokens at a time. These are called segments. Because of this, when we are processing a particular segment, we cannot see the previous segment. This is called "context fragmentation".

Segment-level recurrence

  1. Process one segment of text
  2. Cache its hidden states
  3. Process the next segment with access to the cached states
  4. Reuse this process across later segments
segment 1 -> hidden states -> cached memory
                                   |
segment 2 -------------------------+-> next hidden states

The memory contains representations of earlier text. Attention can use these representations when it processes a later segment.

Relative positional encoding

The original Transformer uses absolute positional encodings. Transformer-XL uses relative positional information.

  • Absolute position describes where a token occurs in the sequence
  • Relative position describes the distance between token positions
  • Relative positions help attention use cached states from earlier segments

Connection to XLNet

XLNet uses Transformer-XL's segment recurrence and relative positional encoding. XLNet adds a permutation language modelling objective.