Linear Classifier

The linear classifier is special case 1 of the Gaussian Bayes classifier. All classes share the same covariance matrix, ${\mathbf{\Sigma }}_i=\mathbf{\Sigma }$. The quadratic term of the Quadratic Classifier then cancels, and the decision boundary becomes a hyperplane.

Main idea

With $\boldsymbol{\Sigma}_i=\boldsymbol{\Sigma}$, $g_i(\mathbf{x})=\mathbf{w}_i^T\mathbf{x}+w_{i0}$. The boundary is a hyperplane, but in general it is not orthogonal to the line between the means.

---

Discriminant Function

With ${\mathbf{\Sigma }}_i=\mathbf{\Sigma }$, the term $-\frac{1}{2}\ln |\mathbf{\Sigma }|$ is the same for all classes and is dropped:

$$\begin{array}{rl}{g}_i(\mathbf{x})& =-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_i{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-{\mathbf{\mu }}_i)+\ln p({\omega }_i)\\ & =-\frac{1}{2}{\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}\mathbf{x}+{\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}\mathbf{x}-\frac{1}{2}{\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_i+\ln p({\omega }_i)\end{array}$$

The term $-\frac{1}{2}{\mathbf{x}}^T{\mathbf{\Sigma }}^{-1}\mathbf{x}$ does not depend on the class label $i$. Drop it:

$$\begin{array}{rl}{g}_i(\mathbf{x})& ={\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}\mathbf{x}-\frac{1}{2}{\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_i+\ln p({\omega }_i)\\ & ={\mathbf{w}}_i^T\mathbf{x}+w_{i0}\end{array}$$

where:

• ${\mathbf{w}}_i={\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_i$ (this uses ${\mathbf{\Sigma }}^{-1}$ symmetric, so ${\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}\mathbf{x}=({\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_i{)}^T\mathbf{x}$)
• $w_{i0}=-\frac{1}{2}{\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_i+\ln p({\omega }_i)$ is the threshold or bias for class ${\omega }_i$

$g_i(\mathbf{x})$ is a linear function of $\mathbf{x}$, so this is a linear classifier.

---

Decision Boundary

$$g_i(\mathbf{x})=g_j(\mathbf{x})\Rightarrow ({\mathbf{w}}_i-{\mathbf{w}}_j{)}^T\mathbf{x}+w_{i0}-w_{j0}=0$$ $$({\mathbf{\mu }}_i-{\mathbf{\mu }}_j{)}^T{\mathbf{\Sigma }}^{-1}\mathbf{x}-\frac{1}{2}({\mathbf{\mu }}_i-{\mathbf{\mu }}_j{)}^T{\mathbf{\Sigma }}^{-1}({\mathbf{\mu }}_i+{\mathbf{\mu }}_j)+\ln [p({\omega }_i)/p({\omega }_j)]=0$$

The middle term uses ${\mathbf{\mu }}_i^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_i-{\mathbf{\mu }}_j^T{\mathbf{\Sigma }}^{-1}{\mathbf{\mu }}_j=({\mathbf{\mu }}_i-{\mathbf{\mu }}_j{)}^T{\mathbf{\Sigma }}^{-1}({\mathbf{\mu }}_i+{\mathbf{\mu }}_j)$, which is true because ${\mathbf{\Sigma }}^{-1}$ is symmetric.

Shape of the boundary:

• a straight line in 2D space $\mathbf{x}$
• a plane in 3D space $\mathbf{x}$
• a hyperplane in higher dimensional space $\mathbf{x}$

---

Orientation of the Hyperplane

The normal vector of the hyperplane is ${\mathbf{\Sigma }}^{-1}({\mathbf{\mu }}_i-{\mathbf{\mu }}_j)$. The line between the means has direction ${\mathbf{\mu }}_i-{\mathbf{\mu }}_j$. In general, ${\mathbf{\Sigma }}^{-1}$ rotates this direction, so the hyperplane is generally not orthogonal to the line between the means.

The figure shows equal but asymmetric Gaussian distributions. The decision hyperplanes are not perpendicular to the line that connects the means. In the right column, the prior changes from 0.5 to 0.9 for ${\omega }_2$. The hyperplane moves away from the mean of ${\omega }_2$, so the region ${\Re }_2$ becomes larger.

In what case is it orthogonal?

The hyperplane is orthogonal to the line between the means when ${\mathbf{\Sigma }}^{-1}({\mathbf{\mu }}_i-{\mathbf{\mu }}_j)$ is parallel to ${\mathbf{\mu }}_i-{\mathbf{\mu }}_j$. The lecture case is special case 2, $\mathbf{\Sigma }={\sigma }^2\mathbf{I}$, in the Minimum Distance Classifier note. (Extra: it is also true when ${\mathbf{\mu }}_i-{\mathbf{\mu }}_j$ is an eigenvector of $\mathbf{\Sigma }$.)