Introduction

The Gaussian score contains a distance between the pattern and the class mean . This distance is the Mahalanobis distance. Here it is compared with the ordinary Euclidean distance. Both work in any number of dimensions, and they differ in how they treat different directions.

Main idea

Euclidean distance is the ruler distance and treats every direction the same. Mahalanobis distance measures the offset in units of the class spread, using the variances and the tilt in $\boldsymbol{\Sigma}_i$. All points on one contour ellipse of a Gaussian have the same Mahalanobis distance to the mean but different Euclidean distances. When $\boldsymbol{\Sigma}_i=\mathbf{I}$ the two are the same, and when $\boldsymbol{\Sigma}_i=\sigma^2\mathbf{I}$ they differ only by the factor $1/\sigma^2$.


Overview

The two distances differ only by the matrix in the middle:


Step 1: definitions

is called the square of the Mahalanobis distance between and . Compare it with the square of the Euclidean distance:

Both are written as squared distances. With the disappears and the two are equal.

With the two differ only by a fixed scale, , so the nearest mean is the same with either. This is why the minimum distance classifier can use the Euclidean distance.


Step 2: 1D case

Reading the figure:

  • the curve is a 1D Gaussian PDF, marked at and
  • in 1D the Mahalanobis distance is the squared number of standard deviations from the mean
  • the points always have , whatever is, and the shaded tails beyond them each hold about 2.3% of the probability (the figure rounds to 2.5%)

So the Mahalanobis distance says how unusual is for the class. It also does not depend on units: changing units scales and by the same factor, which cancels in the ratio.


Step 3: 2D case and contours

Samples drawn from a 2D Gaussian lie in a cloud centred on the mean. The Gaussian density depends on only through , so every set of points with the same has the same density. These sets are contours, lines of equal probability density, and in 2D they are ellipses centred on .

Reading the figure:

  • the left surface is the 2D Gaussian density, a hill centred on the mean
  • the middle plot shows samples (black dots) around the mean , with red contour ellipses: all points on one ellipse have the same Mahalanobis distance to the mean
  • the same points have different Euclidean distances, because the ellipse is closer to the mean along its short axis than along its long axis
  • the right plot shows a larger cloud of samples (red) with one contour ellipse (black) and the mean (star), and the two distance formulas are written over the plots

Step 4: why the spread matters

Take a class with no covariance and different variances, . Then and:

Comparing two points, each at the same ruler distance from the mean:

  • point A is away along , so
  • point B is away along , so
  • if , A is a normal member of the class and B is very unusual, while for both and cannot tell them apart

With a nonzero covariance the same thing happens along the tilted axes of the ellipse: an offset that goes against the tilt of the cloud is far in Mahalanobis distance, even when it is short by the ruler.


Step 5: comparison

PropertyMahalanobis Euclidean
uses the covarianceyes, through no
points at equal distanceellipses (ellipsoids)circles (spheres)
depends on unitsnoyes
classifier that uses itquadratic and linear Gaussian classifiersminimum distance classifier

Why do we need this

A Gaussian classifier scores a pattern by how far it is from each class mean, so how the distance is measured matters. Mahalanobis distance measures "how unusual is this point for this class", which is the question a classifier really asks.


Next step

Quadratic Classifier is the general case, where every class measures distance with its own .


Visual 1: one ruler distance, many Mahalanobis distances

One tilted class centred on its mean:

  • the dashed circle is every point at ruler distance 1, , and the blue ellipse is every point at
  • the ellipse reaches further along the long axis of the cloud and less along the short axis (the eigenvectors of )
  • so the same circle point is near on the long axis, , and far on the short axis,
  • are the variances along those axes, the tilted version of in Step 4

Visual 2: units change the nearest mean

One student, m, kg, with male and female:

  • with height in m, the weight gap dominates , and the nearest mean is
  • with height in cm, the same height gaps become numbers 100 times bigger, and the nearest mean flips to
  • divides each gap by its own , so the units cancel and the answer is either way
  • each pair of bars is scaled on its own, so only compare the two bars inside a pair