Introduction
A few pieces of math come back again and again across Bayes classifiers, density estimation, PCA, LDA, the small-sample fixes, neural networks, CNNs and Transformers. These notes explain what each one means and where it is used, not how to compute it. Each note answers three questions: what is it, what does it mean physically, and which steps of which notes rely on it.
Main idea
The statistical half is about the shape of a data cloud: the covariance matrix describes it, its eigenvectors and eigenvalues give its axes and spreads, the inverse makes it round, the trace adds the spreads up, the rank counts how many directions the samples fill, and the Mahalanobis distance measures offsets in units of the spread. The likelihood and the softmax turn scores into probabilities. The dot product (coordinate or similarity) and the gradient (solve or descend) run through both halves, from the Bayes classifier to the Transformer.
Notes
Read them in this order, since each builds on the ones before:
- Covariance Matrix in Pattern Recognition: the shape of a data cloud, and what the scatter matrices are
- Eigenvectors and Eigenvalues in Pattern Recognition: the axes of the cloud and the spread along each one
- Inverse Covariance and Whitening in Pattern Recognition: what dividing by the spread does, and why tiny eigenvalues are dangerous
- Mahalanobis Distance in Pattern Recognition: distance in units of the cloud's own spread
- Trace in Pattern Recognition: the total spread, and the spread kept by a set of directions
- Rank in Pattern Recognition: how many directions the samples really fill, and why , and keep appearing
- Lagrange Multiplier in Pattern Recognition: why the best direction under a length limit is an eigenvector
- Likelihood in Pattern Recognition: how well an explanation accounts for the data, for classes and for parameters
- Softmax and Posterior in Pattern Recognition: turning scores into probabilities, from the Bayes posterior to attention
- Dot Product and Projection in Pattern Recognition: coordinate along a direction, or similarity to a template
- Gradient in Pattern Recognition: uphill direction, set to zero to solve or followed downhill to train
One line each
How to say each idea in one sentence:
- covariance matrix: a symmetric matrix whose diagonal is the variance of each feature and whose off-diagonal entries say how pairs of features move together, so it describes the size, stretch and tilt of an ellipse-shaped cloud
- eigenvector and eigenvalue of a covariance matrix: an axis of the cloud, and the variance of the data along that axis
- inverse and whitening: the same axes with spreads , which squash the cloud into a round ball, so tiny spreads become huge stretches
- Mahalanobis distance: the squared offset from the mean measured in units of the spread along each axis, the ordinary distance after whitening
- trace: the total variance of all features, the average squared distance from the mean, equal to the sum of the eigenvalues
- rank: the number of independent directions the samples spread in, at most for samples
- Lagrange multiplier: at the best point under a constraint the gradient points straight out of the constraint, which makes PCA's answer an eigenvector
- likelihood: how probable the observed data is under an explanation, a class or a parameter value
- softmax: positive weights that add up to 1, which are exactly the posteriors when the scores are log-probabilities
- dot product: the coordinate of a vector along a direction, or its similarity to a template
- gradient: the direction of fastest increase of the error, set to zero to solve or followed downhill to train
Threads across the topics
Some ideas appear in both halves under different names:
- one direction, three ways: the linear classifier's normal , the two-class LDA direction and the least squares direction with suitable targets are the same vector
- template matching: the minimum distance classifier, a neuron, a convolution filter and an attention score all take a dot product with a template
- probabilities from scores: the Bayes posterior, the normalized Parzen weights, the MLP outputs and the attention weights are all normalized scores
- too few samples: too-small eigenvalues, misleading information, overfitting, and the fixes (adding , weight penalties, weight sharing, PCA) are one problem seen from different sides
Why do we need this
Knowing what each object means turns the formulas of Part 2 into pictures of data clouds, templates and error landscapes. It also lets you explain why a method works, which matters more here than being able to compute the numbers.
Visual 1: one cloud, read by each idea

How to read it, left to right:
- left: gives the tilted ellipse, its eigenvectors are the axes, and each arrow is drawn with length , the spread along that axis
- left: the trace is the total spread counted two ways, , and the black ellipses are points of equal Mahalanobis distance and
- middle: whitening with (a square root of ) turns the same cloud into a round ball with spread 1 on every axis, so the ellipses become circles and is the ordinary squared distance
- right: two samples only fill one line, so the rank is , , and there is nothing to divide by across the line, which is why small samples break the inverse