Bayes Error for Minimizing Risk
Contents
Introduction
In class we discussed Bayes rule for minimizing the probability of error. Our goal is to generalize this rule to minimize risk instead of probability of error. For simplicity we deal with the two class case. Then we provide examples for the cases of 1D and 2D features, and we derive Bayes rule for minimizing risk in these cases.
We briefly motivate this topic by considering the following scenario. Suppose that a doctor has to classify a tumor as cancerous or benign. We consider the cost of misclassifying the tumor as benign to be high, since we would like the patient to be treated quickly if cancer is present. Therefore, we would assign a high cost to misclassifying the tumor as benign. We shall show how to incorporate this cost into Bayes rule in this article.
Bayes rule for minimizing risk
Let $ x \in \mathbb R^n $ be a feature vector. Let $ q_i(x) $ be the posterior probability of class $ i $ (denoted as $ \omega_i $) given $ x $, and let $ P_i $ be the prior probability for $ omega_i $. Let $ p_i(x) $ be the class conditional density for class $ i $. Denote $ c_{ij} $ as the cost of deciding $ x \in \omega_i $ with $ \omega_j $ as the true class. The conditional cost of assigning $ x \in \omega_i $ given $ x $ is
$ r_i(x) = c_{i1} q_1(x) + c_{i2} q_2(x) $
where $ i = 1, 2 $.
We assign $ x $ so that cost is minimum:
$ r_1(x) \lessgtr^{\omega_1}_{\omega_2} r_2(x) $
which says to decide $ \omega_1 $ if $ r_1(x) < r_2(x) $ and $ \omega_2 $ otherwise. If we make the decision this way, the total cost becomes
$ \begin{align} E[r(x)] &= \int \text{min}[ r_1(x), r_2(x) ] p(x) dx \\ &= \int \text{min} [c_{11} q_1(x) + c_{12} q_2(x), c_{21} q_1(x) + c_{22} q_2(x)] p(x) dx \\ &= \int \text{min} [c_{11} P_1 p_1(x) + c_{12} P_2 p_2(x), c_{21} P_1 p_1(x) + c_{22} P_2 p_2(x) ] dx \\ &= \int_{R_1} c_{11} P_1 p_1(x) + c_{12} P_2 p_2(x) dx + \int_{R_2} c_{21} P_1 p_1(x) + c_{22} P_2 p_2(x) dx \\ &= (c_{21} P_1 + c_{22} P_2) + \int_{R_1} (c_{11} - c_{21}) P_1 p_1(x) + (c_{12} - c_{22}) P_2 p_2(x) dx \end{align} $
where $ R_1 $ and $ R_2 $ are partitions of $ \mathbb R^n $, $ R_1 $ and $ R_2 $ are determined by the decision rule from Eq. (-----------), and we use $ \int_{R_2} p_1(x) dx = 1 - \int_{R_1} p_1(x) dx $ in the last line. To minimize Eq. (---------), we make the integrand as negative as possible:
$ (c_{12} - c_{22}) P_2 p_2(x) \lessgtr^{\omega_1}_{\omega_2} (c_{21} - c_{11}) P_1 p_1(x) $
which is equivalent to assigning $ x \in \omega_1 $ if $ x $ makes the integrand negative, and $ x \in \omega_2 $ otherwise. Bayes test for minimum cost can now be stated as
$ \frac{p_1(x)}{p_2(x)} \gtrless^{\omega_1}_{\omega_2} \frac{(c_{12} - c_{22})P_2}{(c_{21} - c_{11})P_1}. $
Example 1: 1D features
Consider two classes and $ x \in \mathbb R $. Let $ p_1(x) = N(\mu_1, \sigma_1) $ and $ p_2(x) = N(\mu_2, \sigma_2) $. For simplicity, let $ c_{11} = c_{22} = 0 $. From Eq. (--------) we assign $ x \in \omega_1 $ if
$ \begin{align} c_{21} P_1 p_1(x) &> c_{12} P_2 p_2(x) \\ \iff \text{ln}(c_{21} P_1) + \text{ln}(p_1(x)) &> \text{ln}(c_{12} P_2) + \text{ln}(p_2(x)) \\ \iff \text{ln}(c_{21} P_1) - \text{ln}(\sigma_1) - \frac{1}{2}\text{ln}(2 \pi) - \frac{(x - \mu_1)^2}{2 \sigma_1^2} &> \text{ln}(c_{12} P_2) - \text{ln}(\sigma_2) - \frac{1}{2}\text{ln}(2 \pi) - \frac{(x - \mu_2)^2}{2 \sigma_2^2} \end{align} $
so the discriminant function becomes
$ \frac{1}{2} \left( \frac{1}{\sigma_2^2} - \frac{1}{\sigma_1^2} \right) x^2 + \left( \frac{\mu_1}{\sigma_1^2} - \frac{\mu_2}{\sigma_2^2} \right) x + \frac{1}{2} \left( \frac{\mu_2^2}{\sigma_2^2} - \frac{\mu_1^2}{\sigma_1^2} \right) + \text{ln} \left( \frac{\sigma_2}{\sigma_1} \right) + \text{ln} \left( \frac{P_1 c_{21}}{P_2 c_{12}} \right) > 0 $
which has the form
$ a x^2 + b x + c > 0 $
where
$ a = \frac{1}{2} \left( \frac{1}{\sigma_2^2} - \frac{1}{\sigma_1^2} \right), $
$ b = \frac{\mu_1}{\sigma_1^2} - \frac{\mu_2}{\sigma_2^2}, $
and
$ c = \frac{1}{2} \left( \frac{\mu_2^2}{\sigma_2^2} - \frac{\mu_1^2}{\sigma_1^2} \right) + \text{ln} \left( \frac{\sigma_2}{\sigma_1} \right) + \text{ln} \left( \frac{P_1 c_{21}}{P_2 c_{12}} \right). $
This form is similar to Bayes rule for minimizing error, except for the factor of $ \text{ln} \left( \frac{P_1 c_{21}}{P_2 c_{12}} \right) $, which shifts the decision thresholds. An equivalent formulation is to decide $ x \in \omega_1 $ if
$ \frac{p_1(x)}{p_2(x)} > \frac{P_2 c_{12}}{P_1 c_{21}} $
or
$ \frac{p_1(x)}{p_2(x)} > \lambda $
where
$ \lambda = \frac{P_2 c_{12}}{P_1 c_{21}}. $
We can interpret the decision rule as a modification of the Neyman-Pearson criterion that takes into account the priors and the cost.
Example 2: 2D features
Let $ x \in \mathbb R^2 $ with normal class conditional densities
$ p_i(x) = \frac{1}{(2 \pi)^n} | \Sigma_i |^{-1/2} \text{exp}\left[ -\frac{1}{2} (x - \mu_i)^T \sigma_i^{-1} (x - \mu_i) \right] $
where $ i = 1, 2 $. For simplicity, let $ c_{11} = c_{22} = 0 $. Similar to Eq. (--------), we decide $ x \in \omega_1 $ if
$ \begin{align} c_{21} P_1 p_1(x) &> c_{12} P_2 p_2(x) \\ \iff \text{ln}(c_{21} P_1) + \text{ln}(p_1(x)) &> \text{ln}(c_{12} P_2) + \text{ln}(p_2(x)) \\ \iff \text{ln}(c_{21} P_1) - \text{ln}|\Sigma_1| - (x - \mu_1)^T \Sigma_1^{-1} (x - \mu_1) &> \text{ln}(c_{12} P_2) - \text{ln} |\Sigma_2| - (x - \mu_2)^T \Sigma_2^{-1} (x - \mu_2) \end{align} $
so the discriminant function becomes
$ \frac{1}{2} x^T (\Sigma_2^{-1} - \Sigma_1^{-1}) x + x^T( \Sigma_1^{-1} \mu_1 - \Sigma_2^{-1} \mu_2 ) + \frac{1}{2} ( \mu_2^T \Sigma_2^{-1} \mu_2 - \mu_1^T \Sigma_1^{-1} \mu_1 ) + \text{ln} \left( \frac{P_1 c_{21}}{P_2 c_{12}} \right) + \frac{1}{2} \text{ln} \left( \left| \frac{\Sigma_2}{\Sigma_1} \right| \right) $
which has the form
$ x^T A x + b^T x + c > 0 $
where
$ A = \frac{1}{2} (\Sigma_2^{-1} - \Sigma_1^{-1}), $
$ b = \Sigma_1^{-1} \mu_1 - \Sigma_2^{-1} \mu_2, $
and
$ c = \frac{1}{2} ( \mu_2^T \Sigma_2^{-1} \mu_2 - \mu_1^T \Sigma_1^{-1} \mu_1 ) + \text{ln} \left( \frac{P_1 c_{21}}{P_2 c_{12}} \right) + \frac{1}{2} \text{ln} \left( \left| \frac{\Sigma_2}{\Sigma_1} \right| \right). $
As in the 1D case, we decide $ x \in \omega_1 $ if
$ \frac{p_1(x)}{p_2(x)} > \lambda $
where
$ \lambda = \frac{P_2 c_{12}}{P_1 c_{21}}. $
When $ \Sigma_1 = \Sigma_2 $, the Bayes classifier becomes a linear discriminant function.
To give a specific illustration, we generate data from 2
classes and take
$ \mu_1 = \left[ \begin{array}{cc} 8 & 1 \end{array} \right]^T $
$ \mu_2 = \left[ \begin{array}{cc} 7 & 7 \end{array} \right]^T $
$ \Sigma = \left[ \begin{array}{cc} 6 & 1 \\ 1 & 6 \end{array} \right]. $
The data is classified using the discriminant function from Eq. (------). To see the effects of $ c_{12} $ and $ c_{21} $, we vary their values and examine how the separating hyperplane shifts in Fig. 1. We examine the following cases:
- When $ c_{12} = 1 $ and $ c_{21} = 1 $, the cost of misclassifying classes 1 and 2 are equal. We are reduced to Bayes rule. The separating hyperplane is positioned to minimize the probability of error, as Fig. 1(a) shows.
- When $ c_{12} = 5 $ and $ c_{21} = 1 $, the cost of misclassifying class 2 is high. Thus, the separating hyperplane shifts toward class 1, so less points from class 2 are misclassified, as Fig. 1(b) shows.
- When $ c_{12} = 1 $ and $ c_{21} = 5 $, the cost of misclassifying class 1 is high, so the separating hyperplane shifts toward class 2. As a result, less points from class 1 are misclassified, as Fig. 1(c) shows.
Summary and Conclusions
In this lecture we have shown that the probability of error ($Prob \left[ Error \right] $) when using Bayes error, is upper bounded by the Chernoff Bound. Therefore,
for $ \beta \in \left[ 0, 1 \right] $.
When $ \beta =\frac{1}{2} $ then $ \varepsilon_{\frac{1}{2}} $ in known as the Bhattacharyya bound.
References
[1]. Duda, Richard O. and Hart, Peter E. and Stork, David G., "Pattern Classication (2nd Edition)," Wiley-Interscience, 2000.
[2]. Mireille Boutin, "ECE662: Statistical Pattern Recognition and Decision Making Processes," Purdue University, Spring 2014.
Questions and comments
If you have any questions, comments, etc. please post them On this page.
