Introduction
A Transformer is built from two stacks: encoders, which read the whole input sequence, and decoders, which write the output sequence one token at a time. This note goes through the overall structure with machine translation, then the parts inside one encoder and one decoder, then how the output is produced.
Main idea
All input tokens enter the encoder stack in parallel. Each encoder has a self-attention layer and a feed-forward layer. Each decoder has the same two plus an encoder-decoder attention layer that reads the final encoder output. The decoder output goes through a linear layer and softmax to classify the next output word, which is fed back in for the next step.
Overview
The data flow, from input sentence to output sentence:
The final encoder output is sent to every decoder in the stack.
Step 1: one box
A machine translation application takes a sentence in one language, for example "Je suis étudiant", and outputs its translation in another, "I am a student".
The main points:
- all words (tokens) of a sentence, or of an image, are input into the Transformer in parallel
- inside, the Transformer is an encoder part followed by a decoder part
- the output is a linear classification of the decoder output: which word comes next
Step 2: stacks of encoders and decoders

Reading the figure:
- the encoder part is a stack of encoders, all with the same structure (each has its own weights)
- the decoder part is a stack of decoders, also all with the same structure
- the input goes up through the encoders one after another
- the output of the final encoder is passed to every decoder, shown by the fan of arrows
Step 3: inside an encoder and a decoder

What they have in common:
- each starts from an embedding of its tokens plus a position encoding
- a self-attention layer, where the tokens of one sequence attend to each other
- a feed-forward layer, applied to each token separately
- after each layer, layer normalization, with a residual connection (shown in Token Embedding and Position Encoding)
How they differ:
- the encoder reads the source sequence (the input), and the decoder reads the target sequence produced so far
- the decoder has an extra encoder-decoder attention layer between its self-attention and its feed-forward layer, which looks at the encoder output
- the decoder's self-attention is masked, so a position cannot look at later words that have not been generated yet
- only the decoder side ends with the linear layer and softmax that pick the next word
Step 4: how the output is produced

Decoding goes one time step at a time:
- time step 1: the encoders process the whole input sentence once, and their final output is turned into the keys and values used by every decoder (how keys and values are built is in Query Key and Value Projections)
- the encoders run only once, and only the decoder repeats at every time step
- the decoder starts from a start token alone and produces the first word, "I"
- time step 2 (the figure): "I" is fed back in as a previous output, and the decoder produces "am"
- this repeats until an end token is produced
- each input word enters as an embedding plus a position signal, labelled "embedding with time signal"
Because each output is fed back in to produce the next one, the Transformer is a generative model.
Why do we need this
It gives the map that the other notes fill in. Every box in it is made from the same few operations: embeddings, linear projections, attention and pointwise feed-forward layers.
Next step
Token Embedding and Position Encoding shows how the input tokens become the vectors that enter the first encoder.
Visual 1: decoding one word per step

Translating "Je suis étudiant" into "I am a student":
- the encoders run once at step 1, and every later step reuses their keys and values
- at each step the decoder reads the start token plus every word produced so far, with the newest one in dark orange
- the decoder output goes through the linear layer and softmax to pick the next word (green), which is fed back in at the next step
- decoding stops when the end token is produced, here after 5 steps (Step 4)
Visual 2: masked self-attention

Five decoder tokens:
- left: every score above the diagonal (, grey) is set to before the softmax, and the scores below it are kept
- right: , so those weights are exactly 0, and each row spreads its weight only over itself and earlier tokens
- the start token can only look at itself, so its single weight is 1
- this is how the decoder's self-attention avoids words that have not been generated yet (Step 3)