| Line 14: | Line 14: | ||
</center> | </center> | ||
---- | ---- | ||
| + | |||
| + | = Introduction = | ||
| + | |||
| + | In the field of machine learning, one big topic is classification problem. A linear classifier is an algorithm that make the classification decision on a new test data point base on a linear combination of the features. | ||
| + | |||
| + | There are two classes of linear classifier: Generative model and Discriminative model: | ||
| + | |||
| + | * The generative model measures the joint distribution of the data and class. | ||
| + | ** Examples are Naive Bayes Classifier, Linear Discriminant Analysis. | ||
| + | |||
| + | * The discriminivative model makes no assumption on the joint distribution of the data. Instead, it takes the data as given and tries to maximize the conditional density (Prob(class|data)) directly. | ||
| + | ** Examples are Logistic Regression, Perceptron and Support Vector Machine. | ||
| + | |||
| + | = Intuition and derivation of Logistic Regression = | ||
| + | |||
| + | Consider a simple classification problem. The goal is to tell whether a person is male or female base on one feature: hair length. The data is given as <math>(x_i,y_i)</math> where i is the index number of the training set, and <math>x_i</math> is hair length in centimeters and <math>y_i=1</math> indicates the person is male and 0 if female. Assume women has longer hair length the distribution of training data will look like this: | ||
| + | |||
| + | [[Image:Example.jpg]] | ||
| + | |||
| + | = Maximum Likelihood Estimation = | ||
| + | |||
| + | = Numerical optimization = | ||
Revision as of 17:42, 13 May 2014
Logistic regression
A slecture by ECE student Borui Chen
Partly based on the ECE662 Spring 2014 lecture material of Prof. Mireille Boutin.
Contents
Introduction
In the field of machine learning, one big topic is classification problem. A linear classifier is an algorithm that make the classification decision on a new test data point base on a linear combination of the features.
There are two classes of linear classifier: Generative model and Discriminative model:
- The generative model measures the joint distribution of the data and class.
- Examples are Naive Bayes Classifier, Linear Discriminant Analysis.
- The discriminivative model makes no assumption on the joint distribution of the data. Instead, it takes the data as given and tries to maximize the conditional density (Prob(class|data)) directly.
- Examples are Logistic Regression, Perceptron and Support Vector Machine.
Intuition and derivation of Logistic Regression
Consider a simple classification problem. The goal is to tell whether a person is male or female base on one feature: hair length. The data is given as $ (x_i,y_i) $ where i is the index number of the training set, and $ x_i $ is hair length in centimeters and $ y_i=1 $ indicates the person is male and 0 if female. Assume women has longer hair length the distribution of training data will look like this:
