Introduction
The dot product is the single operation that runs from the first classifier to the Transformer. It has two readings: the coordinate of a vector along a direction, and the similarity of a vector to a template. Feature extraction uses the first reading, classifiers, neurons, convolution and attention use the second. The projection step it comes from is in Principal Component Analysis.
Main idea
$\mathbf{a}^T\mathbf{b}=\|\mathbf{a}\|\|\mathbf{b}\|\cos\theta$. With a unit vector $\boldsymbol{\phi}$, $\boldsymbol{\phi}^T\mathbf{x}$ is how far $\mathbf{x}$ lies along $\boldsymbol{\phi}$ (projection), which is how PCA and LDA make features. With a template $\mathbf{w}$, $\mathbf{w}^T\mathbf{x}$ is large when $\mathbf{x}$ points the same way as $\mathbf{w}$ (similarity), which is what a linear classifier, a neuron, a convolution filter and an attention score all compute.
What it is
A matrix product is many dot products at once: entry of is row of dotted with column of .
Reading 1: coordinate along a direction
With a unit vector :
- is the coordinate of along , like reading a ruler laid along it
- is the closest point to on that line, and the leftover is perpendicular to it
- with several unit directions as the columns of , lists all the coordinates, and is the closest point in the subspace they span
Reading 2: similarity to a template
With any vector seen as a template:
- is large and positive when points the same way as , zero when perpendicular, negative when opposite
- it also grows with the lengths of both vectors, so long vectors score high even when the match is poor
- the link to distance: , so "closest template" and "largest dot product, corrected for the template's length" pick the same winner
- the set is a flat boundary (hyperplane) perpendicular to
Where it is used
| Note, step | Which reading, and what it means there |
|---|---|
| Visual Data Dimensionality Reduction, Overview | coordinate: each feature is how far the image lies along one learned direction |
| Principal Component Analysis, Step 2 | coordinate: is the one number kept per sample |
| PCA Reconstruction, Step 2 | coordinate: keeps the part inside the kept directions |
| Discriminant Function in Eigenspace, Step 4 | coordinate: along each eigenvector of a class |
| Minimum Distance Classifier, Step 4 | similarity: is template matching with the class mean |
| Linear Classifier | similarity: with a template that has been whitened by |
| Neuron Model | similarity: the weight vector is the pattern the neuron responds to, and is its boundary |
| LMS Gradient Descent Learning | similarity: each update adds a multiple of to , moving the template toward or away from the sample |
| Convolutional Layer | similarity: the filter is dotted with the window at every position, so it is template matching everywhere in the image |
| Query Key and Value Projections | coordinate: each entry of is the token's coordinate along a learned direction |
| Attention as Weighted Sum of Values, Step 1 | similarity: scores how well token matches what token is looking for |
Common traps
Points that are easy to get wrong:
- a projection coordinate needs a unit vector; with a longer vector the number is scaled by its length
- a dot product is not a pure angle: dividing by in attention, and the term in template matching, both correct for size
- the learned directions in are not orthonormal, unlike the PCA directions
Test yourself
Why does the minimum distance classifier become "template matching"?
, and is the same for every class. So the nearest mean is the class with the largest .
What does a convolution filter "look for" in an image?
At each position it takes the dot product with the window under it, which is largest where the window looks like the filter. So it is a template matched at every position.
How is a PCA feature like a neuron's weighted sum?
Both are one dot product of the input with a vector. PCA picks the vector as an eigenvector, a neuron learns it by gradient descent.
Why do we need this
The dot product is the thread that connects every topic of Part 2. Seeing "coordinate" and "similarity" in each formula makes the Transformer look like the same ideas as the Bayes classifier, used again with learned directions.
Visual 1: coordinate along an eigenvector

How to read it:
- the grey points are samples of one class, and the dashed line through the mean is its first eigenvector
- the black arrow is the offset of one sample from the mean
- dropping straight onto the axis lands on the orange point, and how far it sits from the mean along the axis is the feature
- the red dotted leftover is perpendicular to , so it is the part of that this one feature throws away
- a step by step version with numbers is in Projection onto an Axis
Visual 2: similarity to a template

All three vectors have the same length, and only the angle to the template changes:
- small angle: is close to 1, so the score is large and positive
- right angle: , so the score is zero and has nothing in common with
- angle past : , so the score is negative and points against the template
- a neuron, the minimum distance classifier and an attention score all measure this
Visual 3: flat boundary

Here and :
- every point on the black line gives exactly , so the line is the boundary
- sticks out of the line at a right angle and points to the side where the score grows
- the point scores and the point scores , so they fall on opposite sides
- with features the line becomes a flat hyperplane, the boundary of the linear classifier and of a single neuron
Visual 4: convolution as template matching

How to read it:
- the stem plot on the left is the filter , an up then down pattern
- the filter slides along the input, and at each position the output is
- the output peaks where the input holds the same pattern (shaded), and stays near zero on noise
- so a convolution filter is one dot product repeated at every position