Difference between revisions of "User talk:Leedj" - Rhea

Latest revision as of 11:34, 13 April 2014

Click here for PDF version

Introduction

In class we discussed Bayes rule for minimizing the probability of error. Our goal is to generalize this rule to minimize risk instead of probability of error. For simplicity we deal with the two class case. Then we provide examples for the cases of 1D and 2D features, and we derive Bayes rule for minimizing risk in these cases.

We briefly motivate this topic by considering the following scenario. Suppose that a doctor has to classify a tumor as cancerous or benign. We consider the cost of misclassifying the tumor as benign to be high, since we would like the patient to be treated quickly if cancer is present. Therefore, we would assign a high cost to misclassifying the tumor as benign. We shall show how to incorporate this cost into Bayes rule in this article.

Bayes rule for minimizing risk

Let $ x \in \mathbb R^n $ be a feature vector. Let $ q_i(x) $ be the posterior probability of class $ i $ (denoted as $ \omega_i $) given $ x $, and let $ P_i $ be the prior probability for $ \omega_i $. Let $ p_i(x) $ be the class conditional density for class $ i $. Denote $ c_{ij} $ as the cost of deciding $ x \in \omega_i $ with $ \omega_j $ as the true class. The conditional cost of assigning $ x \in \omega_i $ given $ x $ is

where $ i = 1, 2 $.

We assign $ x $ so that cost is minimum:

which says to decide $ \omega_1 $ if $ r_1(x) < r_2(x) $ and $ \omega_2 $ otherwise. If we make the decision this way, the total cost becomes

where $ R_1 $ and $ R_2 $ are partitions of $ \mathbb R^n $, $ R_1 $ and $ R_2 $ are determined by the decision rule from Eq. (2), and we use $ \int_{R_2} p_1(x) dx = 1 - \int_{R_1} p_1(x) dx $ in the last line. To minimize Eq. (3), we make the integrand as negative as possible:

which is equivalent to assigning $ x \in \omega_1 $ if $ x $ makes the integrand negative, and $ x \in \omega_2 $ otherwise. Bayes test for minimum cost can now be stated as

Example 1: 1D features

Consider two classes and $ x \in \mathbb R $. Let $ p_1(x) = N(\mu_1, \sigma_1) $ and $ p_2(x) = N(\mu_2, \sigma_2) $. For simplicity, let $ c_{11} = c_{22} = 0 $. From Eq. (4) we assign $ x \in \omega_1 $ if

so the discriminant function becomes

which has the form

where

and

This form is similar to Bayes rule for minimizing error, except for the factor of $ \text{ln} \left( \frac{P_1 c_{21}}{P_2 c_{12}} \right) $, which shifts the decision thresholds. An equivalent formulation is to decide $ x \in \omega_1 $ if

or

where

We can interpret the decision rule as a modification of the Neyman-Pearson criterion that takes into account the priors and the cost.

Example 2: 2D features

Let $ x \in \mathbb R^2 $ with normal class conditional densities

where $ i = 1, 2 $. For simplicity, let $ c_{11} = c_{22} = 0 $. Similar to Eq. (6), we decide $ x \in \omega_1 $ if

so the discriminant function becomes

which has the form

where

and

As in the 1D case, we decide $ x \in \omega_1 $ if

where

When $ \Sigma_1 = \Sigma_2 $, the Bayes classifier becomes a linear discriminant function.

To give a specific illustration, we generate data from 2 classes and take

Fig 1: Data for class 1 (crosses) and class 2 (circles). In all cases, Prob($ \omega_1 $) = Prob($ \omega_2 $) = 0.5. Misclassified points are shown in red. Values of $ \mu_1 $, $ \mu_2 $, and $ \Sigma $ are given in Eqs. (10) - (12). As the figures show, the separating hyperplanes shift depending on the values of $ c_{12} $ and $ c_{21} $.

The data is classified using the discriminant function from Eq. (9). To see the effects of $ c_{12} $ and $ c_{21} $, we vary their values and examine how the separating hyperplane shifts in Fig. 1. We examine the following cases:

• When $ c_{12} = 1 $ and $ c_{21} = 1 $, the cost of misclassifying classes 1 and 2 are equal. We are reduced to Bayes rule. The separating hyperplane is positioned to minimize the probability of error, as Fig. 1(a) shows.

• When $ c_{12} = 5 $ and $ c_{21} = 1 $, the cost of misclassifying class 2 is high. Thus, the separating hyperplane shifts toward class 1, so less points from class 2 are misclassified, as Fig. 1(b) shows.

• When $ c_{12} = 1 $ and $ c_{21} = 5 $, the cost of misclassifying class 1 is high, so the separating hyperplane shifts toward class 2. As a result, less points from class 1 are misclassified, as Fig. 1(c) shows.

Summary and Conclusions

We have derived Bayes rule for minimizing risk. The rule can be stated as

To illustrate this rule, we have given two examples dealing with 1D and 2D features. For both cases, the separating hyperplane shifts depending on the costs. Figure 1 provides a nice demonstration for the 2D case. When the cost of misclassifying class $ i $ is high, the separating hyperplane shifts to reduce the number of points misclassified from class $ i $. We hope that this material provides the reader with a more general understanding of Bayes rule.

References

[1]. K. Fukunaga, Introduction to Statistical Pattern Recognition (Academic, New York, 1972).

[2]. Mireille Boutin, "ECE662: Statistical Pattern Recognition and Decision Making Processes," Purdue University, Spring 2014.

Questions and comments

If you have any questions, comments, etc. please post them on this page.