Introduction

This note collects the critical components of a Transformer, shows how the Vision Transformer (ViT) uses them on images, compares ViT with a CNN, and ends with the conclusion of the deep learning topics, which ties the MLP, the CNN and the Transformer together.

Main idea

A Transformer is built from three operations: projections $\mathbf{Q}=\mathbf{X}\mathbf{W}_q$, $\mathbf{K}=\mathbf{X}\mathbf{W}_k$, $\mathbf{V}=\mathbf{X}\mathbf{W}_v$, attention $\text{softmax}(\mathbf{Q}\mathbf{K}^T/\sqrt{d_k})\mathbf{V}$, and a pointwise network $\mathbf{Y}=h(h(h(\mathbf{Z}\mathbf{W}_1)\mathbf{W}_2)\mathbf{W}_3)$. The encoder and decoder Transformer is generative. ViT turns an image into patch tokens and uses only the encoder, as a classifier. Its learnable parts all use the concept of convolution, the same weights for every token, and it gathers global information with attention, whose weights are computed from each test input.


Overview

The whole Transformer in three lines, with token rows in () and outputs in ():

Here is the activation, the same as in Pointwise Feed-Forward Network. The figure below writes it as .


Step 1: critical components

Reading the figure:

  • the left part shows the decoder producing one word at a time ("Hi, how are you?") from a start token
  • the right part is the full Transformer: an encoder block repeated (multi-head attention, add and norm, feed forward, add and norm) and a decoder block repeated (masked multi-head attention, multi-head attention over the encoder output, feed forward, each with add and norm), where is the number of layers in the stack, not the number of tokens
  • both sides start from an embedding plus a positional encoding, and the decoder ends with linear and softmax layers that give the output probabilities of the next word

The encoder and decoder Transformer is a generative model: the decoder produces one output token at a time, and feeds each one back in to produce the next.


Step 2: Vision Transformer

How ViT handles an image:

  • split the image into fixed-size patches
  • flatten each patch and apply a linear projection to get one token per patch, the image version of a word embedding, and since the same projection is used for every patch, it is like a convolution whose kernel size and stride both equal the patch size
  • add position embeddings, so the network knows where each patch came from
  • add one extra learnable classification token at the start of the sequence
  • pass the sequence through a standard Transformer encoder (no decoder), repeated times, each with normalization, multi-head attention and a block labelled MLP
  • read the class from the classification token's output with a small MLP head

The question mark on the block labelled "MLP": it is applied to each patch token separately with shared weights, so it is the pointwise feed-forward network, a stack of convolutions, and not an MLP over the whole image.

ViT is used as a classifier, so it is not generative. Its encoder also puts the normalization before each attention and MLP block (Norm, then the block, then add), while Token Embedding and Position Encoding wrote the original order, add first and then normalize.


Step 3: Vision Transformer and CNN

What each side offers:

  • ViT outperforms state-of-the-art convolutional networks on multiple computer vision benchmarks, while needing fewer computational resources to train, after being pre-trained on large amounts of data
  • CNNs have a proven track record in many computer vision tasks and handle large-scale datasets efficiently
  • ViT offers advantages where global dependencies and contextual understanding are crucial

The question mark on "global":

  • "CNN is local, Transformer is global" is not the full answer, since the higher layers of a deep CNN also gather information from a large area
  • the deeper difference is in how information is combined: a CNN uses weights fixed by training, while attention uses weights generated from each test input

Step 4: vision-language Transformer

The same building blocks can join images and text. Referring segmentation means finding and outlining the pixels of the object that a sentence refers to. A vision-language Transformer (VLT) for this task takes an image and a sentence such as "small elephant on the left":

  • it extracts vision features from the image and language features from the sentence, and fuses them
  • it generates query vectors from both the vision and the language features, each focusing on a different part of the description
  • a Transformer encoder and decoder use these queries to find the object the sentence describes
  • a mask decoder outputs the segmentation mask of that object, an image that marks which pixels belong to it

Step 5: conclusion of deep learning

The main points, in order:

  • the success of AI lies in machine learning
  • the success of machine learning lies in learning general rules, and not over-fitting to specific details of the training data (over-fitting means fitting the training data so closely that new inputs are handled worse)
  • this is realized by learning in convolution with the minimum filter size, which is the CNN
  • many convolutional layers are then needed to capture information from a large area, which is deep learning
  • deep learning is made possible by the simplest nonlinear activation function, ReLU
  • every single learned parameter of a CNN goes through all inputs, which greatly improves generalization, but the CNN captures information only through weights learned from the training data, which limits its capability
  • the Transformer captures global information by non-learnable attention modules: the way it combines tokens is not stored as weights learned from the training set, it is computed from the specific test input
  • all learnable modules of the Transformer use the concept of convolution

Why do we need this

It shows that the Transformer is a general architecture: the same projections, attention and pointwise layers work for words, image patches, or both together. It also completes the comparison of the three network families: MLP, CNN and Transformer.


Next step

From CNN to Transformer has the overview and reading order for the whole topic.


Visual 1: one patch becomes one token row

A image cut into 16 patches of pixels, following patch 6:

  • patch 6 is unrolled into one long row of numbers, a vector
  • one shared projection () turns it into the token (), and the same is used for every patch, like a convolution with kernel size and stride both 16
  • the 16 patch tokens plus the classification token (blue) are stacked as the rows of ,
  • the rows no longer say where each patch was, so a position embedding is added to each row before the encoder (Step 2)

Visual 2: fixed weights against weights from the input

How 8 output tokens combine 8 input tokens, darker meaning a larger weight, for two different inputs A and B:

  • top: a CNN layer with filter uses the same band of weights for both inputs, fixed after training, and each output only sees its neighbours
  • bottom: the attention weights softmax can sit on any token, and the pattern changes with the input
  • both attention maps use the same learned and , and only the input changed
  • this is the deeper difference in Step 3: not local against global, but fixed weights against weights computed from each test input