Bayes Decision Theory
Introduction
The Bayesian decision theory is a valuable approach to solve a pattern classification problem. It is based on quantifying the tradeoffs between various classification decisions using the probability of events occurring and the costs that accompany the decisions. Here, we are assuming that the problems are posed in probabilistic terms and all relevant probability values are known (It is important to note that in reality its not always like this).
Consider a situation where we have a stack of cards where each card is either a diamond or spade. We can denote x = x1 for diamonds, and x = x2 for spades. Suppose we want to design a system that will be able to predict the next card that will come up. We also know the prior probability P(x1) that the next card is diamonds, and some prior probability P(x2) that it's spades, and both probabilities sum up to 1 (since we only have two variables). We can therefore use the following decision rule :that if P(x1) > P(x2), then the card is diamonds, otherwise it is spades. How well that works will depend on how much greater P(x1) is. If it is much greater than P(x2) then our decision will favor diamonds most of the time, however if P(x1) = P(x2) then we have only a 50% chance of being correct.
However, in most cases we wont be making decisions with so little information. For example if we had information about the value of color of the shapes on the cards (the value of a color refers to the degree of lightness and darkness of a color), we can describe this as a variable y and we consider y to be a random variable whose distribution depends on the state of the card and is expressed as p(y|x). This is called the class-conditional probability density function, and it is defined as the probability of y given that the state is x. The equation for the conditional probability is given as:
$ P(y|x)= \frac{P(xy)}{P(x)} \qquad\qquad\qquad\qquad\qquad (1) $
and
$ P(x|y)= \frac{P(xy)}{P(y)} \qquad\qquad\qquad\qquad\qquad (2) $
The difference between p(y|x1) and p(y|x2) describes the difference in color values between the diamonds and spades in a stark of cards. Suppose we know both the prior probability P(xj) and the conditional probability p(y|xj) for j = 1,2. If we also measure the color values for the card as y, we can rearrange the equations 1 and 2 to come up with Bayes formula which is:
$ P(x_j|y)= \frac{p(y|x_j)p(x_j)}{P(y)} \qquad\qquad\qquad\qquad\qquad (3) $
the formula above can be expressed as
$ result = \frac{likelihood x prior}{evidence} $
