Introduction
An eigenvector of a matrix is a direction that the matrix does not turn, only stretches, and the eigenvalue is the stretch factor (the general idea is in Eigenvalues and Eigenvectors). In pattern recognition the matrix is almost always a covariance or scatter matrix of data. Then the eigenvectors are the natural axes of the data cloud, and each eigenvalue is how much the data spreads along its axis. Almost every method from PCA to the small-sample fixes is a statement about these axes and spreads.
Main idea
For a covariance matrix $\boldsymbol{\Sigma}$, the unit eigenvectors $\boldsymbol{\phi}_k$ are the perpendicular axes of the data cloud, and the eigenvalue $\lambda_k=\boldsymbol{\phi}_k^T\boldsymbol{\Sigma}\boldsymbol{\phi}_k$ is the variance of the data along $\boldsymbol{\phi}_k$. Along these axes the features are uncorrelated, so an ellipse-shaped cloud is described by $n$ separate spreads, one per axis. A large $\lambda_k$ means the data moves a lot that way, a small one means it barely moves, and a zero one means it never moves.
What it is
In words:
- most vectors are both stretched and turned when multiplied by
- an eigenvector keeps its direction, and only its length changes by the factor
- only the direction matters, so eigenvectors are taken with unit length
What the matrix does to a direction
Why "not turned by " means "axis of the cloud":
- is the average of the offsets, each weighted by how far it lies along
- so points to where the cloud "pulls" the direction
- if is tilted away from the long axis, the points far out along the long axis pull it back toward that axis, so turns away from
- along an axis of the ellipse the pulls on the two sides balance, so there is no sideways pull and stays along
- the same axes are the directions where the spread stops changing as turns: the largest spread, the smallest, and in more dimensions the saddle directions in between (Lagrange Multiplier in Pattern Recognition)
Physical meaning for a covariance matrix
Picture the data cloud as an ellipse (in more dimensions, an ellipsoid). Then:
- eigenvectors are the axes of the ellipse: the longest axis is the eigenvector with the largest eigenvalue, the shortest axis is the one with the smallest
- eigenvalues are the spreads: is the variance of the data projected onto , because is the variance along any unit direction , and for an eigenvector it equals
- the axes are perpendicular: a covariance matrix is symmetric, and the eigenvectors of a symmetric matrix can always be chosen orthonormal
- the axes are the data's own coordinate system: describe each sample by its coordinates along them, and the new features are uncorrelated, with variances ( becomes the diagonal ); uncorrelated is weaker than independent, and the two match only for Gaussian data
- the inverse has the same axes: has the same eigenvectors with eigenvalues , so it shrinks the long axes and stretches the short ones (Inverse Covariance and Whitening in Pattern Recognition)
- the whole matrix is a sum of spreads: , one independent spread per axis
What the size of tells you:
- large: the samples vary a lot along , and with few samples the largest ones also come out a little too large
- small: the samples barely vary along , and with few samples it comes out smaller than the truth, because this direction was picked as the one where these particular samples happen to be tightest
- zero: the samples never move along , so the matrix has no inverse in that direction
What "large spread" is worth depends on the job:
- to represent the data (PCA), large means important: dropping that axis would lose a lot of each sample
- to classify, large can be useless or harmful: it may be the spread inside each class, where the classes overlap
- to estimate reliably (small-sample fixes), large means trustworthy: it is measured well, while the small ones are not
Physical meaning for other matrices
The same idea, a direction plus a number that scores it, appears with other matrices:
- for in LDA, the eigenvector is a projection direction and the eigenvalue is its Fisher ratio, the between-class spread divided by the within-class spread along it
- for a scatter matrix such as or , the meaning is the covariance meaning above, for all samples or for the samples inside their classes
Where it is used
| Note, step | What the eigenvectors and eigenvalues mean there |
|---|---|
| Principal Component Analysis, Steps 5 and 6 | the best single direction to represent the data is the eigenvector of with the largest , and is the variance it keeps |
| PCA Reconstruction, Step 3 | dropping axis loses exactly its variance , so the error is the sum of the dropped eigenvalues |
| PCA Reconstruction, Step 4 | samples give at most nonzero eigenvalues, so the data fills at most directions |
| Eigen-Decomposition, Step 6 | rotating onto the eigenvectors makes the features uncorrelated, with variances |
| Mahalanobis and Euclidean Distance | the contours of equal distance are ellipses whose axes are the eigenvectors |
| Linear Classifier, Step 3 | the boundary is perpendicular to the line of means when lies along an eigenvector of |
| Linear Discriminant Analysis, Steps 3 and 4 | the best directions for separating classes are eigenvectors of , and each is a Fisher ratio |
| LDA Variants, Step 1 | the range space is the eigenvectors with nonzero (where data spreads), the null space is those with zero |
| Discriminant Function in Eigenspace, Steps 5 and 6 | the classifier divides the offset along each axis by , so a too-small training gives that axis a huge, misleading weight |
| Eigenvalue Regularization, PCA for Classification, Eigenspectrum Model | three ways to fix the unreliable small eigenvalues: raise them, drop their axes, or replace them with a model |
Common traps
Points that are easy to get wrong:
- an eigenvector is a direction, not a data point, and is the same axis as
- a large eigenvalue means large spread, not "important for classification": the largest spread can be the direction where the classes overlap (Scatter Matrices, Step 1)
- the eigenvalues of a matrix estimated from samples are not the true ones: the small ones come out too small
- the eigenvectors of are not perpendicular in general: they are perpendicular after the within-class cloud is made round (whitened), and turning back to the original space bends them
- zero within-class spread is read two ways: NDA treats it as perfect separation, while the small-sample notes treat it as an accident of too few samples (LDA Variants, Step 3)
Test yourself
Why does PCA keep the eigenvectors with the largest eigenvalues?
Each eigenvalue is the variance kept along its eigenvector, and the reconstruction error is the variance thrown away. Keeping the largest eigenvalues keeps the most variance and loses the least.
Why are features along the eigenvectors uncorrelated?
In the eigenvector axes the covariance matrix becomes , which is diagonal. Every off-diagonal entry, the covariance between two new features, is zero.
Why does a small eigenvalue hurt a Gaussian classifier trained on few samples?
The classifier divides by . Along a direction where the few training samples happened to be tight, is too small, so is too large, and a new sample of the right class looks very far from its own mean.
What does a zero eigenvalue of say about the training data?
Along that eigenvector, every class's training samples sit at their class mean, with no spread at all. The matrix cannot be inverted there, which is why plain LDA fails with few samples.
Why do we need this
Every Part 2 method that works with covariance or scatter matrices reasons about the axes and spreads of a data cloud. Knowing that "eigenvector = axis, eigenvalue = spread along it" turns each formula into a picture: PCA keeps the long axes, the Mahalanobis distance divides by the spread on each axis, and the small-sample problem is about axes whose spread was measured too small.
Visual 1: directions that the matrix does not turn

In each panel the black arrow is a unit direction and the coloured arrow is :
- left: for a direction off the axes, is turned by toward the long axis of the cloud, the pull of the far-out points
- middle: along the long axis the pulls balance, so only gets longer, by the large factor
- right: along the short axis it only gets shorter, by the small factor
- so the directions not turned by are exactly the axes of the cloud, and the stretch factor is the spread
Visual 2: eigenvalue as the spread along its axis

How to read it:
- left: the dashed lines are the eigenvectors of the sample covariance, and each arrow has length , one spread along its axis
- right: the histograms are the coordinates of the same samples
- the measured variance of is and of is , exactly and
- a large means the data moves a lot along , a small one means it barely moves