Difference between revisions of "Fisher Linear Discriminant OldKiwi" - Rhea

Revision as of 17:36, 25 March 2008

Introduction

Fisher's linear discriminant is a classification method that projects high-dimensional data onto a line and performs classification in this one-dimensional space. The projection maximizes the distance between the means of the two classes while minimizing the variance within each class. See Lecture 10_OldKiwi for detailed explanation.

Finding the separating hyperplane and finding the projection direction are dual problems

Fisher's Linear Discriminant relies on dimension reduction. If we want to project D-dimensional data down to one dimension we can use $ y=\omega^T \vec{x} $ . After that, by determining a threshold $ \omega_0 $, we can select class 1 if $ y>\omega_0 $ and class 2 if $ y<\omega_0 $.

However, dimension reduction can cause considerable amount of information loss leading not being able to separate the data in reduced dimension, while the data is separable in D dimension. Fisher's Linear Discriminant method uses that idea by adjusting the weight vector $ \omega $ for projection to minimize the amount of overlapping data in reduced dimension. Here are some figures from C. M. Bishop's book to understand the idea completely.

Context: Classical Discriminant Analysis_OldKiwi Problem

Given data $ y_1,\cdots, y_d \in \mathbb{R}^n $, from 2 classes. When n is big, it may be difficult to separate classes, because of computational issues.

In this case we try to find a projection $ \pi: R^n \rightarrow R^k, k <n $ s.t . $ \pi(y_1), \cdots, \pi(y_d) $ can be separated.

Derivation of Fisher's Linear Discriminant

Main article: Derivation of Fisher's Linear Discriminant_OldKiwi

References

"Two variations on Fisher's linear discriminant for pattern recognition" is a nice journal article. The paper provides two fast and simple techniques for improving on the classification performance provided by Fisher's linear discriminant for two classes.

See the following link for the detailed information: Two variations on Fisher's linear discriminant for pattern recognition.