Introduction
After attention mixes information across tokens, each encoder and decoder applies a learnable feed-forward network. It looks like an MLP, but it is applied pointwise: the same small network, with the same weights, processes each token's row on its own.
Main idea
$\mathbf{Y}=h(h(h(\mathbf{Z}\mathbf{W}_1)\mathbf{W}_2)\mathbf{W}_3)$, which is $\mathbf{y}_i=h(h(h(\mathbf{z}_i\mathbf{W}_1)\mathbf{W}_2)\mathbf{W}_3)$ for each token row. All tokens share the same weights, so each learnable parameter goes through all inputs. This is the $1\times1$ convolution of a CNN again, and it is not an MLP over the whole sequence.
Overview
The sizes, with tokens and layer widths :
The number of rows stays the whole way: the network changes the features of each token, never the number of tokens.
Step 1: formula
Like the projections , , , the feed-forward network multiplies the token rows by weight matrices, with a nonlinear activation between them:
Here is the attention output and is the activation, acting on each entry separately. This is the general form with three layers. The original Transformer uses two linear layers with ReLU, , in between, and no activation after the last layer:
Step 2: one token at a time
Each row and corresponds to one token. Row of is , and acts entry by entry, so row stays row through every layer:
So token 's output depends only on token 's input. There is no mixing between tokens here, because that was attention's job.
Step 3: meaning of pointwise
All tokens in the inputs and outputs share the same learnable parameters , , . Pointwise means: token 's output depends only on token 's input, and the same weights are used for every token. So each single learnable parameter goes through all inputs.
Compare with an MLP that takes the whole sequence as its input:
- the MLP would flatten all tokens into one vector of numbers
- its first weight matrix would join every feature of every token to every output, which needs about weights for an output as wide as the input, and changes size whenever changes
- each weight would be used for one input position only, so it learns only from that position
The pointwise network instead:
- uses one matrix for all tokens, which needs weights for any
- uses each weight at every token, so each weight is learned from all tokens, which helps generalization in the same way as weight sharing in a CNN
Step 4: 1×1 convolution again
The pointwise network comes again from the key concept of the CNN, the convolution:
- think of the tokens as positions and the features as channels
- a convolution applies the same weights to the channels at every position, with no mixing across positions
- that is exactly for every
- so the feed-forward network is a stack of convolutions, and the same is true of the projections , ,
Where it sits
Each encoder and decoder alternates the two kinds of layer:
- attention mixes information across tokens, with weights set by the input and no learnable combining weights
- the pointwise feed-forward network transforms each token within itself, with learned weights shared by all tokens
- each is wrapped in a residual connection and layer normalization
Why do we need this
Attention only takes weighted sums of value vectors, which is linear in the values. The feed-forward network adds learned, nonlinear processing to each token, while weight sharing keeps it generalizing like a CNN.
Next step
Vision Transformer and CNN puts all the pieces together and applies them to images.
Visual 1: changing one token changes one output row

Four tokens; only the input row is changed, and each cell shows how far that output value moves:
- left: with the pointwise network , only row changes, and the other rows stay exactly the same
- right: an MLP on the whole flattened sequence mixes every token into every output, so all rows change
- so the pointwise network never mixes tokens, which is attention's job (Step 2)
- it is the convolution: the same weights at every position, with no view of the neighbours (Step 4)
Visual 2: weight count against the number of tokens

Widths and as in the original Transformer, two layers, biases left out:
- the pointwise network needs weights, whatever is, so its line stays flat
- an MLP over all tokens with the same widths needs weights, which grows as and reaches at
- the pointwise network also uses each weight at all tokens, so each weight is learned from all of them (Step 3)