Bayes Decision and Classifiers

Part 2 of IE4476 starts with statistical pattern recognition. A computer vision system must decide which class an object belongs to, and it must make this decision under uncertainty. Bayes decision theory gives the optimal rule for this decision and turns it into practical classifiers.

Main idea

Decide the class with the largest posterior probability. This rule gives the smallest probability of error. For Gaussian classes, the rule becomes a quadratic, linear, or minimum distance classifier.

---

Computer Vision System

The system has three stages:

• Preprocessing / normalization takes the raw input $\tilde{\mathbf{x}}$ and gives $\mathbf{x}\in {\mathbb{R}}^n$
• Feature extraction / dimensionality reduction maps $\mathbf{x}$ to a feature vector $\mathbf{f}\in {\mathbb{R}}^d$
• Classification gives the output $\mathbf{y}\in \{0,1{\}}^c$ for $c$ classes, for example 01 for apple and 10 for pear
• two knowledge sources (Type 1 and Type 2) feed the stages

Visual object recognition is very easy for the human brain but very difficult for a machine. This topic studies the last stage, classification.

---

How the Concepts Connect

flowchart TD
    A["MAP Decision Rule<br/>decide the largest p(ωi∣x)"] -->|"how good is the rule?"| B["Probability of Error<br/>average p(ek∣x) over x"]
    A -->|"remove p(x), take ln"| C["Discriminant Functions<br/>gi(x) = ln p(x∣ωi) + ln p(ωi)"]
    C -->|"p(x∣ωi) is Gaussian"| D["Quadratic Classifier<br/>different Σi"]
    C -->|"distance inside gi(x)"| M["Mahalanobis and<br/>Euclidean Distance"]
    D -->|"Special case 1:<br/>Σi = Σ"| E["Linear Classifier<br/>boundary is a hyperplane"]
    E -->|"Special case 2:<br/>Σi = σ²I"| F["Linear Classifier<br/>hyperplane orthogonal to the line of means"]
    F -->|"equal priors"| G["Minimum Distance Classifier<br/>nearest mean"]

Notes in this folder:

• MAP Decision Rule builds the optimal rule from a simple example
• Probability of Error measures the performance of a classifier
• Discriminant Functions turn the decision into a comparison of functions $g_i(\mathbf{x})$
• Mahalanobis and Euclidean Distance compares the two distances that appear in $g_i(\mathbf{x})$
• Quadratic Classifier is the general Gaussian case
• Linear Classifier is special case 1, with one shared covariance matrix
• Minimum Distance Classifier is special case 2 with equal priors

---

Course Content of Part 2

1. MAP/Bayes Decision Rule and Classifiers
2. General Statistical Estimation and Machine Learning
3. Visual Data Dimensionality Reduction
4. Dimensionality Reduction to Enhance Generalization of Machine Learning
5. Traditional Neural Network MLP
6. Convolutional Neural Network CNN
7. From Convolutional Neural Network CNN to Current Powerful Transformer

Part 2 assessment:

• assignment or project is 20%
• final exam is 30%
• prerequisites are probability theory and linear algebra