Introduction

The dot product is the single operation that runs from the first classifier to the Transformer. It has two readings: the coordinate of a vector along a direction, and the similarity of a vector to a template. Feature extraction uses the first reading, classifiers, neurons, convolution and attention use the second. The projection step it comes from is in Principal Component Analysis.

Main idea

$\mathbf{a}^T\mathbf{b}=\|\mathbf{a}\|\|\mathbf{b}\|\cos\theta$. With a unit vector $\boldsymbol{\phi}$, $\boldsymbol{\phi}^T\mathbf{x}$ is how far $\mathbf{x}$ lies along $\boldsymbol{\phi}$ (projection), which is how PCA and LDA make features. With a template $\mathbf{w}$, $\mathbf{w}^T\mathbf{x}$ is large when $\mathbf{x}$ points the same way as $\mathbf{w}$ (similarity), which is what a linear classifier, a neuron, a convolution filter and an attention score all compute.


What it is

A matrix product is many dot products at once: entry of is row of dotted with column of .


Reading 1: coordinate along a direction

With a unit vector :

  • is the coordinate of along , like reading a ruler laid along it
  • is the closest point to on that line, and the leftover is perpendicular to it
  • with several unit directions as the columns of , lists all the coordinates, and is the closest point in the subspace they span

Reading 2: similarity to a template

With any vector seen as a template:

  • is large and positive when points the same way as , zero when perpendicular, negative when opposite
  • it also grows with the lengths of both vectors, so long vectors score high even when the match is poor
  • the link to distance: , so "closest template" and "largest dot product, corrected for the template's length" pick the same winner
  • the set is a flat boundary (hyperplane) perpendicular to

Where it is used

Note, stepWhich reading, and what it means there
Visual Data Dimensionality Reduction, Overviewcoordinate: each feature is how far the image lies along one learned direction
Principal Component Analysis, Step 2coordinate: is the one number kept per sample
PCA Reconstruction, Step 2coordinate: keeps the part inside the kept directions
Discriminant Function in Eigenspace, Step 4coordinate: along each eigenvector of a class
Minimum Distance Classifier, Step 4similarity: is template matching with the class mean
Linear Classifiersimilarity: with a template that has been whitened by
Neuron Modelsimilarity: the weight vector is the pattern the neuron responds to, and is its boundary
LMS Gradient Descent Learningsimilarity: each update adds a multiple of to , moving the template toward or away from the sample
Convolutional Layersimilarity: the filter is dotted with the window at every position, so it is template matching everywhere in the image
Query Key and Value Projectionscoordinate: each entry of is the token's coordinate along a learned direction
Attention as Weighted Sum of Values, Step 1similarity: scores how well token matches what token is looking for

Common traps

Points that are easy to get wrong:

  • a projection coordinate needs a unit vector; with a longer vector the number is scaled by its length
  • a dot product is not a pure angle: dividing by in attention, and the term in template matching, both correct for size
  • the learned directions in are not orthonormal, unlike the PCA directions

Test yourself


Why do we need this

The dot product is the thread that connects every topic of Part 2. Seeing "coordinate" and "similarity" in each formula makes the Transformer look like the same ideas as the Bayes classifier, used again with learned directions.


Visual 1: coordinate along an eigenvector

How to read it:

  • the grey points are samples of one class, and the dashed line through the mean is its first eigenvector
  • the black arrow is the offset of one sample from the mean
  • dropping straight onto the axis lands on the orange point, and how far it sits from the mean along the axis is the feature
  • the red dotted leftover is perpendicular to , so it is the part of that this one feature throws away
  • a step by step version with numbers is in Projection onto an Axis

Visual 2: similarity to a template

All three vectors have the same length, and only the angle to the template changes:

  • small angle: is close to 1, so the score is large and positive
  • right angle: , so the score is zero and has nothing in common with
  • angle past : , so the score is negative and points against the template
  • a neuron, the minimum distance classifier and an attention score all measure this

Visual 3: flat boundary

Here and :

  • every point on the black line gives exactly , so the line is the boundary
  • sticks out of the line at a right angle and points to the side where the score grows
  • the point scores and the point scores , so they fall on opposite sides
  • with features the line becomes a flat hyperplane, the boundary of the linear classifier and of a single neuron

Visual 4: convolution as template matching

How to read it:

  • the stem plot on the left is the filter , an up then down pattern
  • the filter slides along the input, and at each position the output is
  • the output peaks where the input holds the same pattern (shaded), and stays near zero on noise
  • so a convolution filter is one dot product repeated at every position