Discriminant Functions For The Normal Density - Part 2

       Continuing from where we left of in Part 1, in a problem with feature vector y and state of nature variable w, we can represent the discriminant function as:

we will now look at the multiple cases for a multivariate normal distribution.

Case 1: Σ_i = σ²I

       This is the simplest case and it occurs when the features are statistically independent and each feature has the same variance, σ². Here, the covariance matrix is diagonal since its simply σ² times the identity matrix I. This means that each sample falls into equal sized clusters that are centered about their respective mean vectors. The computation of the determinant and the inverse |Σ_i| = σ^2d and Σ_i^-1 = (1/σ²)I. Because both |Σ_i| and the (d/2) ln 2π term in the equation above are independent of i, we can ignore them and thus we obtain this simplified discriminant function:

where ||.|| denotes the Euclidean norm, that is,

       If the prior probabilities are not equal, then the discriminant function shows that the squared distance ||x - μ||² must be normalized by the variance σ² and offset by adding ln P(w_i); therefore if x is equally near two different mean vectors, the optimal decision will favor the priori more likely. Expansion of the quadratic form (x - μ_i)^t(x - μ_i) yields :

which looks like a quadratic function of x. However, the quadratic term x^tx is the same for all i, meaning it can be ignored since it just an additive constant, thereby we obtain the equivalent discriminant function:

where

and

w_i0 is the threshold or bias for the ith category.

       A classifier that uses linear discriminants is called a linear machine. For a linear machine, the decision surfaces for a linear machine are just pieces of hyperplanes defined by the linear equations g_i(x) = g_j(x) for the two categories with the highest posterior probabilities. In this situation, the equation can be written as

where

and

       The equations define a hyperplane through the point x₀ and orthogonal to the vector w. Because w = μ_i - μ_j, the hyperplane separating R_i and R_j is orthogonal to the line linking the means. if P(w_i) = P(w_j), the point x₀ is halfway between the means and the hyperplane is the perpendicular bisector of the line between the means in fig1 below. If P(w_i) ≠ P(w_j), the point x₀ shifts away from the more likely mean.

Figure 1

Case 2: Σ_i = Σ

       Another case occurs when the covariance matrices for all the classes are identical. It corresponds to a situation where the samples fall into hyperellipsoidal clusters of equal size and shape, with the cluster of the ith class being centered around the mean vector μ_i. Both |Σ_i| and the (d/2) ln 2π terms can also be ignored as done in the first step because they are independent of i. This leads to the simplified discriminant function:

       If the prior probabilities P(w_i) are equal for all classes, then the ln P(w_i) term can be ignored, however if they are unequal then the decision will be biased in favor of the more likely priori. Expansion of the quadratic form ('x - μ_i)^tΣ^-1(x - μ_i) results in a sum involving the term x^tΣ^-1'x which is independent of i. After this term is dropped, we get the resulting linear discriminant function:

where

and

       Because the discriminants are linear, the resulting decision boundaries are again hyperplanes. If R_i and R_j are very close, the boundary between them has the equation:

where

and

       Because w = σ^-1(μ_i - μ_j) is generally not in the direction of μ_i - μ_j, the hyperplane separating R_i and R_j is generally not orthogonal to the line between the means. If the prior probabilities are equal, it intersects the line at point x₀, and then x₀ is halfway between the means. If the prior probabilities are not equal, the boundary hyperplane is shifted away from the more likely mean. Figure 2 below shows what the boundary decision looks like for this case

Figure 2

Case 3: Σ_i = arbitrary

       In the general multivariate Gaussian case where the covariance matrices are different for each class, the only term that can be dropped from the initial discriminant function is the (d/2) ln 2π term. The resulting discriminant term is;

where

and

This leads to hyperquadric decision boundaries as seen in the figure below.

. Figure 3

EXAMPLE

       Given the set of data below of a distribution with two classes w₁ and w₂ both with prior probablility of 0.5, find the discriminant functions and decision boundary.

From the data given above we know to use the equations from case 1 since all points in each class have the same variance, therefore the means are:

and the variances are

The discriminant functions are then

and the decision boundary x₀ is going to be halfway between the means at 0.492 because they have the same prior probability.

• Back to Tosin's Honors Project
• Back to ECE 302 Spring 2013. Prof Boutin
• Back to ECE 302