Due Tuesday February 19, 2008
Guidelines
Write a short report to present your results. Be sure to include all the relevant graphs as well as a copy of your code. Teamwork is encouraged, but the write up of your report must be your own. Please write the names of your collaborators on the cover page of your report.
Question 1
Design and execute an experiment that illustrates the Central Limit Theorem. (You may use problem 5 in DHS p. 80 for inspiration.)
Question 2
Consider n-dimensional feature vectors coming from two classes. Assume that the distributions of the feature vectors for the two classes are (known) normal distributions and that the priors for the classes P(w1) and P(w2) are also known. Write a computer program that classifies the feature vectors according to Bayes decision rule. Generate some artificial (normally distributed) data, and test your program on the data you generated. Try feature vectors of various dimensions. Quantify the accuracy of your results. How does the dimension of the feature vectors affect accuracy? (You may use problem 2 in DHS p. 80 for inspiration.)
Question 3
Take a subset of the data you used for Question 2. Use maximum likelihood estimation to estimate the parameters of the feature distribution. Experiment to illustrate the accuracy of the classifier obtained with this estimate. Then repeat the experiments using approximately Gaussian data generated using your answer in Question 1.
Question 4
Replace the words “maximum likelihood estimation” by “Bayesian parameter estimation” in Question 3.
Here is a tutorial on Generating Gaussian Samples_OldKiwi, although the Matlab command normrnd accomplishes this as well.
Helps
- Illustrating the Central Limit Theorem and dice-rolling experiment -- jungtag.gong.1
- `Illustrating the central limit theorem <http://www.stat.wvu.edu/SRS/Modules/CLT/clt.html>`_ : This web site gives basic principles, illustrating the centeral limit theorem when the underlying distribution is normal or chi-square or uniform, bowtie, right wedge, left wedge, and triangular and exercises.
- `Central limit theorem demo applet using simulated dice-rolling experiments <http://www.stat.sc.edu/~west/javahtml/CLT.html>`_ : This applet demonstrates the central limit theorem using simulated dice-rolling experiments.
- `Central Limit Theorem explanation and applet <http://www.math.csusb.edu/faculty/stanton/probstat/clt.html>`_ : introducing the central limit theorem and showing applet to demonstrate the central limit theorem using simulated dice rolloing experiments
- It's a method to generate test data with multivariate normal distribution using randn, sqrtm, and repmat in Matlab** -- jungtag.gong.1
- RANDN(N,Dim) * SQRTM(covar) + REPMAT(mean, N, 1);
, where N is the number of data, Dim is the dimension of feature vecotrs.
- RANDN : Normally distributed random numbers : ex) RANDN(N) is an N-by-N matrix with random entries, chosen from a normal distribution with mean zero, variance one and standard deviation one.
- SQRTM : Matrix square root : ex) X = SQRTM(A) is the principal square root of the matrix A, i.e. X*X = A.
- REPMAT : Replicate and tile an array : ex) B = repmat(A,M,N) creates a large matrix B consisting of an M-by-N tiling of copies of A.
Central Limit Theorem - Galton's Experiment --jin-young.kim.1, Fri, 08 Feb 2008 23:50:57 -0500
1. The Central Limit Theorem says that for large values of the parameter n, the binomial distribution is approximately normal. Galton used the quincunx to show this.
Here is a interesting video to show the experiment http://inter.heao.han.nl/opleidingen/rstnet/rst2/Applets/galton%20board/galtonintro.htm
2. Application of CLT to signal processing Signals can be smoothed by applying a Gaussian filter, which is just the convolution of a signal with an appropriately scaled Gaussian function. Due to the central limit theorem this smoothing can be approximated by several filter steps that can be computed much faster, like the simple moving average.
Ref: "http://en.wikipedia.org/wiki/Central_limit_theorem"
A question regarding Homework:
When we initialise Random Vectors How do we do so for a given covariance matrix ? Will it be good enough if I use the randn function ro generate a vecort of n-gaussian random variables ? can this be called a random vector ? BAsically my question is how do we simulate gaussian data whcih has a NON ZERO elemts in places OTHER than the main diagonal ?
From jin-young.kim.1 Sun Feb 10 20:20:11 -0500 2008
From: jin-young.kim.1
Date: Sun, 10 Feb 2008 20:20:11 -0500
Subject: The Central Limit Theorem vs. The Law of Large Number
Message-ID: <20080210202011-0500@https://engineering.purdue.edu>
=> The Central Limit Theorem contains more information than the Law of Large Numbers, because it gives us detailed information about the shape of the distribution of Sn.
=> For large n the shape is approximately the same as the shape of the standard normal density. More specically, the Central Limit Theorem says that if we standardize and height-correct the distribution of Sn, then the normal density function is a very good approximation to this distribution when n is large.
=> Thus, we have a computable approximation for the distribution for Sn, which provides us with a powerful technique for generating answers for all sorts of questions about sums of independent random variables, even if the individual random variables have different distributions.
Ref: "Intorduction to Probability," by Charles M. Grinstead and J. Laurie Snell
From marc.bosch.ruiz.1 Sun Feb 10 21:18:12 -0500 2008
From: marc.bosch.ruiz.1
Date: Sun, 10 Feb 2008 21:18:12 -0500
Subject: Central Limit Theorem's link (includes a nice applet)
Message-ID: <20080210211812-0500@https://engineering.purdue.edu>
`Central Limit Theorem applet <http://www.math.csusb.edu/faculty/stanton/probstat/clt.html>`_
From stephen.r.rudolph.1 Sun Feb 10 21:38:45 -0500 2008 From: stephen.r.rudolph.1 Date: Sun, 10 Feb 2008 21:38:45 -0500 Subject: Useful Matlab Functions Message-ID: <20080210213845-0500@https://engineering.purdue.edu>
The function "normrnd" will generate an array of random, normal ([Gaussian distribution]) values. To use normrnd, you can pass it 3 parameters. The first parameter is the mean, the second is the [standard deviation], and the third is a tuple specifying the array dimensions. Here is an example for a 10 by 1 matrix of random normal values based on a mean of 100 and a standard deviation of 20:
normrnd(100, 20, [10 1])
ans =
105.8882 73.2764 114.2865 132.4712 86.1645 117.1599 125.0800 68.1254 71.1807 111.4230
From ashish.kamra.1 Sun Feb 10 22:59:46 -0500 2008
From: ashish.kamra.1
Date: Sun, 10 Feb 2008 22:59:46 -0500
Subject: Sampling from Multivariate Normal
Message-ID: <20080210225946-0500@https://engineering.purdue.edu>
For those looking for a routine to sample points from a multivariate normal density, this library of C functions may come in handy ..
http://www.netlib.org/random/ranlib.c.tar.gz
From ashish.kamra.1 Fri Feb 15 09:04:42 -0500 2008
From: ashish.kamra.1
Date: Fri, 15 Feb 2008 09:04:42 -0500
Subject: Hw 1 Problem 2 doubt ?
Message-ID: <20080215090442-0500@https://engineering.purdue.edu>
How are the test samples supposed to be generated ? If I generate them using the distribution from class 1, then they will classified as class by the bayes classifier unless I choose the distributions of both the classes very close to each other. So essentially the classification accuracy will depend upon how well separated the classes are ? Is that correct ? Or am I missing something here ...
From josiah.yoder.1 Sat Feb 16 12:35:08 -0500 2008 From: josiah.yoder.1 Date: Sat, 16 Feb 2008 12:35:08 -0500 Subject: Hw 1 Problem 2 doubt ? Message-ID: <20080216123508-0500@https://engineering.purdue.edu>
Yes. That is correct.
From yamini.nimmagadda.1 Sun Feb 17 13:03:32 -0500 2008 From: yamini.nimmagadda.1 Date: Sun, 17 Feb 2008 13:03:32 -0500 Subject: HW 1 Problem 3 & 4 doubts? Message-ID: <20080217130332-0500@https://engineering.purdue.edu>
- problem 3**: How can we estimate parameters using MLE from the data generated using approximate Gaussian distribution (from question 1) for 2 categories ?? In question 1 we get a single distribution which can have only one set of parameters that belong to one class. But it is required to estimate parameters for 2 classes in question 3.
From jin-young.kim.1 Sun Feb 17 14:35:42 -0500 2008 From: jin-young.kim.1 Date: Sun, 17 Feb 2008 14:35:42 -0500 Subject: Re) HW 1 Problem 3 & 4 doubts? Message-ID: <20080217143542-0500@https://engineering.purdue.edu> In-Reply-To: <20080217130332-0500@https://engineering.purdue.edu>
I think we just need to separately estimate parameters for each class. Finding parameters is done in training stage, so I think we have alreay known to which classes the data belongs.
- problem 4**: For the BPE, what should we assume for the parameters of the distribution of mean? What about the variance?
From jin-young.kim.1 Sun Feb 17 14:26:48 -0500 2008 From: jin-young.kim.1 Date: Sun, 17 Feb 2008 14:26:48 -0500 Subject: HW1 - 2) How to easily compute p(x/w)*Prior(w) using Matlab Message-ID: <20080217142648-0500@https://engineering.purdue.edu>
To classify data using Bayesian classifier we already know Prior(w) and need to compute p(x/w). When p is multidimensioanl Gaussian, we can use Matlab internal function "mvnpdf".
Ex) mvnpdf(X,Mean,Cov)
X <= data we want to classify Mean <= already known when created Cov <= already known when created
To classify data compute pdfgauss and multiply by Prior(w) for each class and choose a class which shows maximum value
From jin-young.kim.1 Sun Feb 17 15:45:54 -0500 2008
From: jin-young.kim.1
Date: Sun, 17 Feb 2008 15:45:54 -0500
Subject: Generate Multivariate Gaussianl RV
Message-ID: <20080217154554-0500@https://engineering.purdue.edu>
I upload this again since somebody deleted my previous post(Feb/12/2008)
This is an example to generate multivariate Gaussian RV using Matlab. It will generate two classes with the specified mean and covariance.
%% Dimension = 2 %% Mean and Covariance mu1 = [1 -1]'; sigma1 = [.9 .4; .4 .3]; mu2 = [0.5 1.5]'; sigma2 = [.9 .4; .4 .3]; mu = [ mu1; mu2];
%% Generate multivariate Gaussian rv X1 = mvnrnd(mu1,sigma1,N1); %% N1xdim X2 = mvnrnd(mu2,sigma2,N2); %% N2xdim
X1, X2 are 2-dim Gaussian with the correspoinding mean and covariance.
From josiah.yoder.1 Tue Feb 19 10:50:12 -0500 2008 From: josiah.yoder.1 Date: Tue, 19 Feb 2008 10:50:12 -0500 Subject: Generate Multivariate Gaussianl RV Message-ID: <20080219105012-0500@https://engineering.purdue.edu> In-Reply-To: <20080217154554-0500@https://engineering.purdue.edu>
Sorry, jin-young, I moved your contribution to [HW: Matlab resources] while cleaning up. I'll be sure to make a clear link next time.
From ka.ki.ng.1 Sun Feb 17 16:54:20 -0500 2008
From: ka.ki.ng.1
Date: Sun, 17 Feb 2008 16:54:20 -0500
Subject: HW 1 Problem 3 & 4 doubts?
Message-ID: <20080217165420-0500@https://engineering.purdue.edu>
In-Reply-To: <20080217130332-0500@https://engineering.purdue.edu>
I just a subset of data from one class and do a MLE on it. Then I do classification on the remaining data using the MLE results obtained. I do this class 1 and class 2 separately.
From neil.d.bedwell.1 Mon Feb 18 00:20:08 -0500 2008 From: neil.d.bedwell.1 Date: Mon, 18 Feb 2008 00:20:08 -0500 Subject: MLE Question Message-ID: <20080218002008-0500@https://engineering.purdue.edu>
After obtaining means and covariances from a subset of data from question 2, do we use the prior probabilities to classify the data?
In other words, do we know Pw1 and Pw2 for classification using our MLE estimates?
From josiah.yoder.1 Tue Feb 19 10:56:10 -0500 2008 From: josiah.yoder.1 Date: Tue, 19 Feb 2008 10:56:10 -0500 Subject: MLE Question Message-ID: <20080219105610-0500@https://engineering.purdue.edu> In-Reply-To: <20080218002008-0500@https://engineering.purdue.edu>
You can use MLE to estimate them, too. But it is reasonable to just assume Pw1 = Pw2.
From d.lunga.1 Wed Feb 20 17:37:05 -0500 2008 From: wadzanai.d.lunga.1 Date: Wed, 20 Feb 2008 17:37:05 -0500 Subject: MATLAB SCRIPTS Message-ID: <20080220173705-0500@https://engineering.purdue.edu>
Please lets keep the H/W discussions and postings in a more clean representation. If you have a matlab or c script please upload it on the seperate page labelled software rather than posting in on this discussion page. I think that way we can easily search and access the scripts. label what the code does to make it it easy for searching.
Thank you.
