Introduction
Before any formula, it helps to see how a person makes a good decision with very little information. This note starts from that everyday guess, writes it as a rule, and finds how often the rule is wrong. Every later note in Bayes Decision and Classifiers grows out of this rule.
Main idea
With no measurement, the best decision is the class with the largest probability, $\omega_k=\arg\max_{\omega_i}p(\omega_i)$. It can still be wrong, with error $p(e)=1-p(\omega_k)$, but no other decision has a smaller error. This is the **maximum probability decision**.
Overview
The note follows one thought:
Step 1: the decision problem
You see a student in a classroom of the School of EEE, but not clearly. You must decide if the student is male () or female ().
Most people decide male, because they know the student population of the school:
- in the School of EEE, and
- in the School of Accounting the shares are the other way round, and , so in an Accounting classroom the same person would decide female
The decision changes with the school because the probabilities change, even though nothing about the student was seen. The question is how to write this decision rule down, and how to show it is a good one.
Step 2: maximum probability decision
is the probability of class before anything about the object is known. The instinctive rule is to pick the class with the larger probability:
For classes this is : pick the class whose probability is the largest.
Step 3: error of the decision
The decision is not always correct. If you decide , you are right whenever the object really is from , which happens with probability . So you are wrong with probability:
Comparing the two possible decisions for two classes:
- decide : wrong whenever the object is from , so
- decide : wrong whenever the object is from , so
- picking the class with the larger probability leaves the smaller one as the error, so its error is the minimum
- for classes, deciding gives , which is smallest when is the largest
In the School of EEE, deciding male gives , and deciding female gives . The decision can still be wrong, but there is no better decision.
Step 4: what a measurement adds
A rule that uses only gives the same answer for every object. It calls every student in the EEE classroom male.
A measurement of the object, such as the student's height, lets the answer change from object to object. The next question is how the probabilities change once is known, and which rule to use then.
Why do we need this
It fixes what "best" means for the rest of the topic: the rule with the smallest probability of error. It also gives the baseline that any measurement has to beat.
Next step
MAP Decision Rule uses the same rule after a measurement is known.
Visual 1: error against the prior

How to read it:
- the two dashed lines are the errors of always deciding and always deciding , as the prior changes
- the black line is the maximum probability decision, which always sits on the lower dashed line, so no fixed choice beats it
- the error is largest, , when both classes are equally common, because the prior then says nothing
- this is from Step 3, drawn for every possible prior
Visual 2: what a measurement adds

School of EEE, with the two classes drawn as over height:
- left: with no measurement the rule decides for every height, so the whole female curve is error,
- right: once the height is known, the decision can change at a threshold, and only the overlap of the two curves is still wrong,
- the red area shrinks because short students are now called female, which is the gain Step 4 asks for