Introduction

An eigenvector of a matrix is a direction that the matrix does not turn, only stretches, and the eigenvalue is the stretch factor (the general idea is in Eigenvalues and Eigenvectors). In pattern recognition the matrix is almost always a covariance or scatter matrix of data. Then the eigenvectors are the natural axes of the data cloud, and each eigenvalue is how much the data spreads along its axis. Almost every method from PCA to the small-sample fixes is a statement about these axes and spreads.

Main idea

For a covariance matrix $\boldsymbol{\Sigma}$, the unit eigenvectors $\boldsymbol{\phi}_k$ are the perpendicular axes of the data cloud, and the eigenvalue $\lambda_k=\boldsymbol{\phi}_k^T\boldsymbol{\Sigma}\boldsymbol{\phi}_k$ is the variance of the data along $\boldsymbol{\phi}_k$. Along these axes the features are uncorrelated, so an ellipse-shaped cloud is described by $n$ separate spreads, one per axis. A large $\lambda_k$ means the data moves a lot that way, a small one means it barely moves, and a zero one means it never moves.


What it is

In words:

  • most vectors are both stretched and turned when multiplied by
  • an eigenvector keeps its direction, and only its length changes by the factor
  • only the direction matters, so eigenvectors are taken with unit length

What the matrix does to a direction

Why "not turned by " means "axis of the cloud":

  • is the average of the offsets, each weighted by how far it lies along
  • so points to where the cloud "pulls" the direction
  • if is tilted away from the long axis, the points far out along the long axis pull it back toward that axis, so turns away from
  • along an axis of the ellipse the pulls on the two sides balance, so there is no sideways pull and stays along
  • the same axes are the directions where the spread stops changing as turns: the largest spread, the smallest, and in more dimensions the saddle directions in between (Lagrange Multiplier in Pattern Recognition)

Physical meaning for a covariance matrix

Picture the data cloud as an ellipse (in more dimensions, an ellipsoid). Then:

  • eigenvectors are the axes of the ellipse: the longest axis is the eigenvector with the largest eigenvalue, the shortest axis is the one with the smallest
  • eigenvalues are the spreads: is the variance of the data projected onto , because is the variance along any unit direction , and for an eigenvector it equals
  • the axes are perpendicular: a covariance matrix is symmetric, and the eigenvectors of a symmetric matrix can always be chosen orthonormal
  • the axes are the data's own coordinate system: describe each sample by its coordinates along them, and the new features are uncorrelated, with variances ( becomes the diagonal ); uncorrelated is weaker than independent, and the two match only for Gaussian data
  • the inverse has the same axes: has the same eigenvectors with eigenvalues , so it shrinks the long axes and stretches the short ones (Inverse Covariance and Whitening in Pattern Recognition)
  • the whole matrix is a sum of spreads: , one independent spread per axis

What the size of tells you:

  • large: the samples vary a lot along , and with few samples the largest ones also come out a little too large
  • small: the samples barely vary along , and with few samples it comes out smaller than the truth, because this direction was picked as the one where these particular samples happen to be tightest
  • zero: the samples never move along , so the matrix has no inverse in that direction

What "large spread" is worth depends on the job:

  • to represent the data (PCA), large means important: dropping that axis would lose a lot of each sample
  • to classify, large can be useless or harmful: it may be the spread inside each class, where the classes overlap
  • to estimate reliably (small-sample fixes), large means trustworthy: it is measured well, while the small ones are not

Physical meaning for other matrices

The same idea, a direction plus a number that scores it, appears with other matrices:

  • for in LDA, the eigenvector is a projection direction and the eigenvalue is its Fisher ratio, the between-class spread divided by the within-class spread along it
  • for a scatter matrix such as or , the meaning is the covariance meaning above, for all samples or for the samples inside their classes

Where it is used

Note, stepWhat the eigenvectors and eigenvalues mean there
Principal Component Analysis, Steps 5 and 6the best single direction to represent the data is the eigenvector of with the largest , and is the variance it keeps
PCA Reconstruction, Step 3dropping axis loses exactly its variance , so the error is the sum of the dropped eigenvalues
PCA Reconstruction, Step 4 samples give at most nonzero eigenvalues, so the data fills at most directions
Eigen-Decomposition, Step 6rotating onto the eigenvectors makes the features uncorrelated, with variances
Mahalanobis and Euclidean Distancethe contours of equal distance are ellipses whose axes are the eigenvectors
Linear Classifier, Step 3the boundary is perpendicular to the line of means when lies along an eigenvector of
Linear Discriminant Analysis, Steps 3 and 4the best directions for separating classes are eigenvectors of , and each is a Fisher ratio
LDA Variants, Step 1the range space is the eigenvectors with nonzero (where data spreads), the null space is those with zero
Discriminant Function in Eigenspace, Steps 5 and 6the classifier divides the offset along each axis by , so a too-small training gives that axis a huge, misleading weight
Eigenvalue Regularization, PCA for Classification, Eigenspectrum Modelthree ways to fix the unreliable small eigenvalues: raise them, drop their axes, or replace them with a model

Common traps

Points that are easy to get wrong:

  • an eigenvector is a direction, not a data point, and is the same axis as
  • a large eigenvalue means large spread, not "important for classification": the largest spread can be the direction where the classes overlap (Scatter Matrices, Step 1)
  • the eigenvalues of a matrix estimated from samples are not the true ones: the small ones come out too small
  • the eigenvectors of are not perpendicular in general: they are perpendicular after the within-class cloud is made round (whitened), and turning back to the original space bends them
  • zero within-class spread is read two ways: NDA treats it as perfect separation, while the small-sample notes treat it as an accident of too few samples (LDA Variants, Step 3)

Test yourself


Why do we need this

Every Part 2 method that works with covariance or scatter matrices reasons about the axes and spreads of a data cloud. Knowing that "eigenvector = axis, eigenvalue = spread along it" turns each formula into a picture: PCA keeps the long axes, the Mahalanobis distance divides by the spread on each axis, and the small-sample problem is about axes whose spread was measured too small.


Visual 1: directions that the matrix does not turn

In each panel the black arrow is a unit direction and the coloured arrow is :

  • left: for a direction off the axes, is turned by toward the long axis of the cloud, the pull of the far-out points
  • middle: along the long axis the pulls balance, so only gets longer, by the large factor
  • right: along the short axis it only gets shorter, by the small factor
  • so the directions not turned by are exactly the axes of the cloud, and the stretch factor is the spread

Visual 2: eigenvalue as the spread along its axis

How to read it:

  • left: the dashed lines are the eigenvectors of the sample covariance, and each arrow has length , one spread along its axis
  • right: the histograms are the coordinates of the same samples
  • the measured variance of is and of is , exactly and
  • a large means the data moves a lot along , a small one means it barely moves