Quadratic Classifier

The quadratic classifier is the Bayes classifier for Gaussian classes in the general case, where each class has its own covariance matrix ${\mathbf{\Sigma }}_i$. Its Discriminant Functions are quadratic in $\mathbf{x}$, so its decision boundaries are curved.

Main idea

With different $\boldsymbol{\Sigma}_i$, only the $\frac{d}{2}\ln 2\pi$ term can be dropped. $g_i(\mathbf{x})$ stays quadratic and the decision surfaces are hyperquadrics.

---

Discriminant Function

For $p(\mathbf{x}|{\omega }_i)=N({\mathbf{\mu }}_i,{\mathbf{\Sigma }}_i)$:

$$g_i(\mathbf{x})=-\frac{1}{2}(\mathbf{x}-{\mathbf{\mu }}_i{)}^T{\mathbf{\Sigma }}_i^{-1}(\mathbf{x}-{\mathbf{\mu }}_i)+\ln p({\omega }_i)-\frac{1}{2}\ln |{\mathbf{\Sigma }}_i|$$

Expand the quadratic term:

$$g_i(\mathbf{x})={\mathbf{x}}^T{\mathbf{W}}_i\mathbf{x}+{\mathbf{w}}_i^T\mathbf{x}+w_{i0}$$

where:

$${\mathbf{W}}_i=-\frac{1}{2}{\mathbf{\Sigma }}_i^{-1},{\mathbf{w}}_i={\mathbf{\Sigma }}_i^{-1}{\mathbf{\mu }}_i$$ $$w_{i0}=-\frac{1}{2}{\mathbf{\mu }}_i^T{\mathbf{\Sigma }}_i^{-1}{\mathbf{\mu }}_i-\frac{1}{2}\ln |{\mathbf{\Sigma }}_i|+\ln p({\omega }_i)$$

Slide notation

The textbook inset on slide 21 writes the bias as ${\omega }_{i0}$ and uses ${\mathbf{x}}^t$ and $P({\omega }_i)$. These notes use the lecturer's notation $w_{i0}$, ${\mathbf{x}}^T$ and $p({\omega }_i)$. The ${\omega }_{i0}$ in the inset is a weight, not a class.

Why the quadratic term stays:

• the term ${\mathbf{x}}^T{\mathbf{W}}_i\mathbf{x}$ depends on ${\mathbf{\Sigma }}_i$, so it is different for each class
• it does not cancel in $g_i(\mathbf{x})-g_j(\mathbf{x})$

---

Decision Boundaries

The boundary between class $i$ and class $j$ is:

$$g_i(\mathbf{x})=g_j(\mathbf{x})$$

The decision surfaces are hyperquadrics. They can be:

• hyperplanes or pairs of hyperplanes
• hyperspheres or hyperellipsoids
• hyperparaboloids
• hyperhyperboloids of various types

Examples in 2 dimensions with 2 classes:

Complex boundaries for a 3-dimensional, 2-class case and a 2-dimensional, multiclass case:

In the multiclass figure, each decision region ${\Re }_i$ is where $g_i(\mathbf{x})$ is the largest. A region does not need to be connected, and one class can have a region inside the region of another class.