Introduction
The Transformer is a neural network that turns an input sequence into an output sequence, such as a sentence into its translation, or a series of speech or time samples into text. It was developed for natural language processing (NLP), and it is now also a powerful architecture for computer vision, often stronger than a CNN. Its power comes from the attention module, which relates every token in a sequence to every other token.
Main idea
Each token becomes one row of a matrix $\mathbf{X}$ ($N$ tokens by $K$ features). Learnable projections give queries, keys and values, $\mathbf{Q}=\mathbf{X}\mathbf{W}_q$, $\mathbf{K}=\mathbf{X}\mathbf{W}_k$, $\mathbf{V}=\mathbf{X}\mathbf{W}_v$. Attention outputs $\mathbf{z}_i=\sum_j(\mathbf{q}_i\cdot\mathbf{k}_j)\,\mathbf{v}_j$, a weighted sum of all tokens, where the combining weights are computed from each input at run time, not stored as fixed learned numbers. A pointwise feed-forward network then processes each token. Every learnable part is shared by all tokens, which is the idea of a $1\times1$ convolution.
Overview
One encoder layer is a short chain of matrix operations. With tokens and features per token:
What each arrow is:
- the first arrow is learnable: three weight matrices shared by all tokens
- the second and third arrows are attention: every token is compared with every other token ( scores), and each output token is a weighted sum of all value tokens
- attention has no weights of its own: its combining weights are computed from and for each new input, and only the projections around it are learned
- the last arrow is learnable: a small network applied to each token row with the same weights
Reminders of terms from the CNN notes:
- a convolution applies the same small weight matrix at every position, to mix the features (channels) of that position only
- ReLU is the activation
- generalization means doing well on new inputs that were not in the training data
- a test input is a new input given to the network after training
Step 1: what a Transformer does
The Transformer has an architecture that transforms an input sequence into an output sequence, such as speech, text and time series. All tokens of the input are fed in at the same time.
Step 2: why not a recurrent network
The background:
- recurrent neural networks (RNNs) and their further development, long short-term memory (LSTM), are clearly suited to sequences and lists, because they read them one step at a time
- the Transformer outperforms them by doing those sequential operations in parallel
Step 3: attention
The Transformer gets its power from its attention module. It captures the relationship between each word or token in a sequence and every other word or token. The original paper of the Transformer is "Attention Is All You Need".
Questions to keep in mind while reading:
- how does the Transformer work, and is attention really everything?
- is it related to the CNN?
- can we simply say the Transformer is better than a CNN because a CNN captures only local information while a Transformer captures global information?
Notation
Conventions:
- a token is a row vector , and the tokens are stacked as the rows of
- so a weight matrix multiplies from the right,
- the earlier MLP notes used column feature vectors with the weight matrix on the left,
Symbols:
- is the number of tokens
- (plain italic) is the number of features per token
- bold is the key matrix
Reading order
The notes follow one line of thought:
- Encoder and Decoder Stacks: the overall structure, and how the output sequence is produced step by step
- Token Embedding and Position Encoding: how each token becomes a row of , and how each layer is wrapped with a residual connection
- Query Key and Value Projections: where attention sits, multiple heads, and the three learnable projections
- Attention as Weighted Sum of Values: scores, weighted sum, softmax, and why attention is not learnable
- Pointwise Feed-Forward Network: the learnable network applied to each token, and why it is pointwise
- Vision Transformer and CNN: the critical components, the Vision Transformer, and the conclusion of deep learning
Concept map
flowchart TD A["Encoder and decoder stacks"] --> B["Embedding + position encoding<br/>X (N × K), residual + layer norm"] B --> C["Projections<br/>Q = XWq, K = XWk, V = XWv"] C --> D["Attention<br/>Z = softmax(QKᵀ/√dk)V<br/>zi = Σ (qi·kj) vj"] D --> E["Pointwise feed-forward<br/>Y = h(h(h(ZW1)W2)W3)"] E --> F["Vision Transformer<br/>and conclusion"]
Why do we need this
The Transformer is the architecture behind modern language models and many top vision models. Seeing it through the CNN (shared weights, convolutions) makes its parts much easier to understand.
Key math ideas
The math ideas this topic relies on, and where each one appears:
- Dot product and projection: each entry of is a coordinate along a learned direction in Query Key and Value Projections, and is a match score in Attention as Weighted Sum of Values
- Softmax and posterior: the attention weights in Attention as Weighted Sum of Values, and the output probabilities in Encoder and Decoder Stacks and Vision Transformer and CNN
- Mahalanobis distance: the score has the same vector, matrix, vector form, with a learned middle matrix, in Attention as Weighted Sum of Values
- Gradient: dividing by keeps the softmax away from regions where the gradient is almost zero, in Attention as Weighted Sum of Values
Visual 1: one encoder layer drawn to scale

Shapes for tokens with features, each cell one number:
- every matrix has rows, one per token, so each token keeps its own row through the whole layer
- green arrows are learned and shared by all tokens: the projections and the pointwise network
- orange arrows are attention: the weights compare every pair of tokens, and each row adds up to 1 (the numbers come from one random input)
- only the attention step mixes tokens, and it has no weights of its own; the reading order follows this chain from top to bottom