Discriminant Functions

A discriminant function $g_i(\mathbf{x})$ gives a score for each class ${\omega }_i$. The classifier picks the class with the largest score. This is the practical form of the MAP Decision Rule.

Main idea

$g_i(\mathbf{x})=\ln p(\mathbf{x}|\omega_i)+\ln p(\omega_i)$. Decide the class with the largest $g_i(\mathbf{x})$. The boundary between class $i$ and class $j$ is $g_i(\mathbf{x})=g_j(\mathbf{x})$.

---

From MAP to Discriminant Functions

The Bayesian (optimal) classification maximises the posterior probability and so minimises the classification error:

$$\text{Decide: }{\omega }_k=\arg \underset{{\omega }_i}{\max }[p({\omega }_i|\mathbf{x})]=\arg \underset{{\omega }_i}{\min }[p(e_i|\mathbf{x})]$$

Since:

$$p({\omega }_i|\mathbf{x})=p({\omega }_i)p(\mathbf{x}|{\omega }_i)p^{-1}(\mathbf{x})$$

and $p(\mathbf{x})$ is not a function of ${\omega }_i$, the classifier only needs to evaluate:

$$g_i(\mathbf{x})=\ln p(\mathbf{x}|{\omega }_i)+\ln p({\omega }_i)$$

and find the class ${\omega }_i$ with the maximum value of $g_i(\mathbf{x})$ for a given pattern $\mathbf{x}$.

Why take the natural logarithm:

• $\ln$ is monotonically increasing, so it does not change which class is the largest
• it simplifies the evaluation when $p(\mathbf{x}|{\omega }_i)$ is an exponential function, such as a Gaussian

flowchart LR
    X["Pattern x"] --> G1["g1(x)"]
    X --> G2["g2(x)"]
    X --> Gc["gc(x)"]
    G1 --> M["Select the<br/>maximum"]
    G2 --> M
    Gc --> M
    M --> D["Decide ωk"]

---

Multivariate Gaussian Class-Conditional PDF

Let $p(\mathbf{x}|{\omega }_i)=N({\mathbf{\mu }}_i,{\mathbf{\Sigma }}_i)$ in $d$ dimensions:

$$p(\mathbf{x}|{\omega }_i)=\frac{1}{(2\pi {)}^{d/2}|{\mathbf{\Sigma }}_i{|}^{1/2}}\exp \left[-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_i{)}^T{\mathbf{\Sigma }}_i^{-1}(\mathbf{x}-{\mathbf{\mu }}_i)\right]$$

Take $\ln$ and add $\ln p({\omega }_i)$. The discriminant function becomes a quadratic function of $\mathbf{x}$:

$$\begin{array}{rl}{g}_i(\mathbf{x})& =-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_i{)}^T{\mathbf{\Sigma }}_i^{-1}(\mathbf{x}-{\mathbf{\mu }}_i)+\ln p({\omega }_i)-\frac{1}{2}\ln |{\mathbf{\Sigma }}_i|-\frac{d}{2}\ln 2\pi \\ & =-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_i{)}^T{\mathbf{\Sigma }}_i^{-1}(\mathbf{x}-{\mathbf{\mu }}_i)+b_i\\ & =-\frac{1}{2}{d}_{{\Sigma }_i}(\mathbf{x},{\mathbf{\mu }}_i)+b_i\end{array}$$

where:

• $d_{{\Sigma }_i}(\mathbf{x},{\mathbf{\mu }}_i)=(\mathbf{x}-{\mathbf{\mu }}_i{)}^T{\mathbf{\Sigma }}_i^{-1}(\mathbf{x}-{\mathbf{\mu }}_i)$ is the square of the Mahalanobis distance
• $b_i=\ln p({\omega }_i)-\frac{1}{2}\ln |{\mathbf{\Sigma }}_i|$
• the term $\frac{d}{2}\ln 2\pi$ is the same for all classes, so it is dropped

So the classifier prefers the class whose mean is closest in Mahalanobis distance, adjusted by the bias $b_i$.

---

Decision Boundary

The decision boundary between class $i$ and class $j$ is where the two scores are equal:

$$g_i(\mathbf{x})=g_j(\mathbf{x})$$

The shape of this boundary depends on the covariance matrices:

Covariance	$g_i(\mathbf{x})$	Boundary	Note
Different ${\mathbf{\Sigma }}_i$	quadratic in $\mathbf{x}$	hyperquadric	Quadratic Classifier
Same ${\mathbf{\Sigma }}_i=\mathbf{\Sigma }$	linear in $\mathbf{x}$	hyperplane	Linear Classifier
Same ${\mathbf{\Sigma }}_i={\sigma }^2\mathbf{I}$	linear in $\mathbf{x}$	hyperplane orthogonal to the line of means	Minimum Distance Classifier (equal priors)