Introduction
The linear classifier is special case 1 of the Gaussian Bayes classifier: all classes share the same covariance matrix, . Every class then measures distance in the same way, the quadratic part of the score cancels, and the boundary between two classes becomes a hyperplane.
Main idea
With $\boldsymbol{\Sigma}_i=\boldsymbol{\Sigma}$, $g_i(\mathbf{x})=\mathbf{w}_i^T\mathbf{x}+w_{i0}$ with $\mathbf{w}_i=\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i$ and $w_{i0}=-\frac{1}{2}\boldsymbol{\mu}_i^T\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i+\ln p(\omega_i)$. The boundary is a hyperplane with normal $\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_i-\boldsymbol{\mu}_j)$, so in general it is not perpendicular to the line between the means.
Overview
Step 1: drop what every class shares
With , the term is the same for all classes and is dropped. Expand the distance as in Quadratic Classifier:
The term does not depend on the class label , because every class uses the same . Drop it:
where:
- , since is symmetric and so
- is called the threshold or bias for class
is a linear function of , so this is a linear classifier.
With equal priors, is also the same for every class, and the first line gives . So the rule is: assign to the class whose mean is nearest in Mahalanobis distance, using the shared .
Step 2: decision boundary
Put in and :
The middle term uses : multiply out the right side, and the two cross terms and are equal (symmetric ) and cancel.
The equation is linear in , so the boundary is:
- a straight line in 2D space
- a plane in 3D space
- a hyperplane in higher dimensional space
Where it sits:
- with equal priors, , and makes the first two terms cancel, so the boundary passes through the midpoint of the means
- with , at the midpoint the first two terms still cancel and the left side equals , so the midpoint already belongs to and the tie point lies further toward , which means the boundary moves away from and the more common class gets a larger region
Step 3: hyperplane orientation
The normal vector of a hyperplane is , the direction it faces. Here:
The line between the means has direction . Multiplying by generally turns this direction, so the decision hyperplane is generally not orthogonal to the line between the means. It tilts to follow the shape of the classes.

Reading the figure:
- both classes are Gaussian with the same tilted (asymmetric) covariance, drawn as density surfaces in 2D and ellipsoids in 3D
- the decision hyperplanes are not perpendicular to the line connecting the means
- in the right-hand panels is raised from to and lowered to , and the hyperplane moves away from the mean of , so grows
When is it orthogonal?
When is parallel to . This is true for special case 2, . It is also true when happens to be an eigenvector of , because and have the same eigenvectors.
Two links to later topics:
- why it tilts: whitens the classes (makes their clouds round), and in that round space the best boundary is the perpendicular bisector of the means; turned back to the original space, that perpendicular tilts against the stretch of the clouds
- the normal is the same vector as the two-class LDA direction in Linear Discriminant Analysis, since is the shared class covariance
Step 4: when to use it
Use the linear classifier when the covariance matrices of the classes are the same, or very close. Compare the estimated of each class first, because this tells you if the shared covariance assumption holds.
It also needs fewer numbers than the quadratic classifier: one for all classes. To estimate it, subtract each class's own mean from its own samples, then put all these offsets into one covariance formula, with the total number of samples:
Pooling the raw samples without first removing each class mean would add the gap between the means to the covariance.
Why do we need this
Many real classes differ mainly in their position and much less in their shape. A shared gives a simple, fast rule, a straight boundary, and a more reliable estimate when samples are few.
Next step
Minimum Distance Classifier adds the assumption , which makes the boundary perpendicular to the line of means, and with equal priors it gives the nearest-mean rule.
Visual 1: whitening explains the tilt

Two classes with one shared, tilted :
- left: the boundary passes through the midpoint of the means, but it is not at a right angle to the dashed line between them
- its normal is turned away from
- right: after both clouds are round, and the same boundary is the perpendicular bisector of the means
- so the tilt in the original space is just the right angle of the round space, stretched back, as in Step 3
Visual 2: pooling the shared covariance

Both classes have the true shape , , :
- left: putting raw samples into one covariance measures the gap between the two clouds, so and the tilt even turns positive
- right: subtracting each class's own mean first stacks both clouds on zero, and the estimate is close to the truth
- this is the formula of Step 4, where each is paired with of its own class