(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
Hw assignment 1
+
__NOTOC__
  
−
Due Tuesday February 19, 2006
+
Due '''Tuesday February 19, 2008'''
−
 
+
−
 
+
−
 
+
−
Guidelines:
+
  
 +
==Guidelines==
 
Write a short report to present your results.
 
Write a short report to present your results.
 
Be sure to include all the relevant graphs as well as a copy of your code.
 
Be sure to include all the relevant graphs as well as a copy of your code.
 
Teamwork is encouraged, but the write up of your report must be your own.
 
Teamwork is encouraged, but the write up of your report must be your own.
 
Please write the names of your collaborators on the cover page of your report.
 
Please write the names of your collaborators on the cover page of your report.
 +
 +
== Question 1 ==
 +
Design and execute an experiment that illustrates the Central Limit Theorem. (You may use problem 5 in DHS p. 80 for inspiration.)
 +
 +
==Question 2==
 +
Consider n-dimensional feature vectors coming from two classes. Assume that the distributions of the feature vectors for the two classes are (known) normal distributions and that the priors for the classes P(w1) and P(w2) are also known. Write a computer program that classifies the feature vectors according to Bayes decision rule. Generate some artificial (normally distributed) data, and test your program on the data you generated. Try feature vectors of various dimensions. Quantify the accuracy of your results. How does the dimension of the feature vectors affect accuracy? (You may use problem 2 in DHS p. 80 for inspiration.)
 +
 +
==Question 3==
 +
Take a subset of the data you used for Question 2. Use maximum likelihood estimation to estimate the parameters of the feature distribution. Experiment to illustrate the accuracy of the classifier obtained with this estimate. Then repeat the experiments using approximately Gaussian data generated using your answer in Question 1.
 +
 +
==Question 4==
 +
Replace the words “maximum likelihood estimation” by “Bayesian parameter estimation” in Question 3.
 +
 +
Here is a tutorial on [[Generating Gaussian Samples_OldKiwi]], although the Matlab command ''normrnd'' accomplishes this as well.
 +
 +
 +
 +
==Discussions==
 +
'''Illustrating the Central Limit Theorem and dice-rolling experiment''' -- jungtag.gong.1
 +
 +
- `Illustrating the central limit theorem <http://www.stat.wvu.edu/SRS/Modules/CLT/clt.html>`_ : This web site gives basic principles, illustrating the centeral limit theorem when the underlying distribution is normal or chi-square or uniform,  bowtie, right wedge, left wedge, and triangular and exercises.
 +
 +
- `Central limit theorem demo applet using simulated dice-rolling experiments <http://www.stat.sc.edu/~west/javahtml/CLT.html>`_  :  This applet demonstrates the central limit theorem using simulated dice-rolling experiments.
 +
 +
- `Central Limit Theorem explanation and applet <http://www.math.csusb.edu/faculty/stanton/probstat/clt.html>`_ : introducing the central limit theorem and showing applet to demonstrate the central limit theorem using simulated dice rolloing experiments
 +
 +
It's a method to generate test data with multivariate normal distribution using randn, sqrtm, and repmat in Matlab**  -- jungtag.gong.1
 +
:  RANDN(N,Dim) * SQRTM(covar) + REPMAT(mean, N, 1);
 +
, where N is the number of data, Dim is the dimension of feature vecotrs.
 +
 +
- RANDN : Normally distributed random numbers : ex) RANDN(N) is an N-by-N matrix with random entries, chosen from a normal distribution with mean zero, variance one and standard deviation one.
 +
 +
- SQRTM : Matrix square root : ex) X = SQRTM(A) is the principal square root of the matrix A, i.e. X*X = A.
 +
 +
- REPMAT : Replicate and tile an array : ex) B = repmat(A,M,N) creates a large matrix B consisting of an M-by-N tiling of copies of A.
 +
 +
 +
'''Central Limit Theorem - Galton's Experiment''' --jin-young.kim.1, Fri, 08 Feb 2008 23:50:57 -0500
 +
 +
1. The Central Limit Theorem says that for large values of the parameter n, the binomial distribution is approximately normal. Galton used the quincunx to show this.
 +
 +
Here is a interesting video to show the experiment
 +
http://inter.heao.han.nl/opleidingen/rstnet/rst2/Applets/galton%20board/galtonintro.htm
 +
 +
2. Application of CLT to signal processing
 +
Signals can be smoothed by applying a Gaussian filter, which is just the convolution of a signal with an appropriately scaled Gaussian function. Due to the central limit theorem this smoothing can be approximated by several filter steps that can be computed much faster, like the simple moving average.
 +
 +
Ref: "http://en.wikipedia.org/wiki/Central_limit_theorem"
 +
 +
A question regarding Homework:
 +
 +
When we initialise Random Vectors How do we do so for a given covariance matrix ? Will it be good enough if I use the randn function ro generate a vecort of n-gaussian random variables ? can this be called a random vector ?  BAsically my question is how do we simulate gaussian data whcih has a NON ZERO elemts in places OTHER than the main diagonal ?
 +
 +
 +
'''The Central Limit Theorem vs. The Law of Large Number''' --jin-young.kim.1, Sun, 10 Feb 2008 20:20:11 -0500
 +
 +
=> The Central Limit Theorem contains more information than the Law of Large Numbers, because it gives us detailed information about the shape of the distribution of Sn.
 +
 +
=> For large n the shape is approximately the same as the shape of the standard normal density. More specically, the Central Limit Theorem says that if we standardize and height-correct the distribution of Sn, then the normal density function is a very good approximation to this distribution when n is large.
 +
 +
=> Thus, we have a computable approximation for the distribution for Sn,
 +
which provides us with a powerful technique for generating answers for all sorts
 +
of questions about sums of independent random variables, even if the individual
 +
random variables have different distributions.
 +
 +
Ref: "Intorduction to Probability," by Charles M. Grinstead and J. Laurie Snell
 +
 +
 +
'''Central Limit Theorem's link (includes a nice applet)''' --marc.bosch.ruiz.1, Sun, 10 Feb 2008 21:18:12 -0500
 +
 +
`Central Limit Theorem applet <http://www.math.csusb.edu/faculty/stanton/probstat/clt.html>`_
 +
 +
 +
'''Useful Matlab Functions''' --stephen.r.rudolph.1, Sun, 10 Feb 2008 21:38:45 -0500
 +
 +
The function "normrnd" will generate an array of random, normal ([Gaussian distribution]) values. To use normrnd, you can pass it 3 parameters. The first parameter is the mean, the second is the [standard deviation], and the third is a tuple specifying the array dimensions. Here is an example for a 10 by 1 matrix of random normal values based on a mean of 100 and a standard deviation of 20:
 +
 +
 +
normrnd(100, 20, [10 1])
 +
 +
ans =
 +
 +
105.8882
 +
73.2764
 +
114.2865
 +
132.4712
 +
86.1645
 +
117.1599
 +
125.0800
 +
68.1254
 +
71.1807
 +
111.4230
 +
 +
 +
'''Sampling from Multivariate Normal''' --ashish.kamra.1, Sun, 10 Feb 2008 22:59:46 -0500
 +
 +
For those looking for a routine to sample points from a multivariate normal density, this library of C functions may come in handy ..
 +
 +
http://www.netlib.org/random/ranlib.c.tar.gz
 +
 +
 +
'''Hw 1 Problem 2 doubt ?''' --ashish.kamra.1, Fri, 15 Feb 2008 09:04:42 -0500
 +
 +
How are the test samples supposed to be generated ? If I generate them using the distribution from class 1, then they will classified as class by the bayes classifier unless I choose the distributions of both the classes very close to each other. So essentially the classification accuracy will depend upon how well separated the classes are ? Is that correct ? Or am I missing something here ...
 +
 +
'''Hw 1 Problem 2 doubt ?''' --josiah.yoder.1, Sat, 16 Feb 2008 12:35:08 -0500
 +
 +
Yes.  That is correct.
 +
 +
 +
'''HW 1 Problem 3 & 4 doubts?''' --yamini.nimmagadda.1, Sun, 17 Feb 2008 13:03:32 -0500
 +
 +
**problem 3**: How can we estimate parameters using MLE from the data generated using approximate Gaussian distribution (from question 1) for 2 categories ?? In question 1 we get a single distribution which can have only one set of parameters that belong to one class. But it is required to estimate parameters for 2 classes in question 3.
 +
 +
'''Re) HW 1 Problem 3 & 4 doubts?''' --jin-young.kim.1, Sun, 17 Feb 2008 14:35:42 -0500
 +
 +
I think we just need to separately estimate parameters for each class. Finding parameters is done in training stage, so I think we have alreay known to which classes the data belongs.
 +
 +
 +
'''problem 4''': For the BPE, what should we assume for the parameters of the distribution of mean? What about the variance?
 +
 +
 +
'''HW1 - 2) How to easily compute p(x/w)*Prior(w) using Matlab''' --jin-young.kim.1, Sun, 17 Feb 2008 14:26:48 -0500
 +
 +
To classify data using Bayesian classifier we already know Prior(w) and need to compute p(x/w). When p is multidimensioanl Gaussian, we can use Matlab internal function "mvnpdf".
 +
 +
Ex) mvnpdf(X,Mean,Cov)
 +
 +
X <= data we want to classify
 +
Mean <= already known when created
 +
Cov  <= already known when created
 +
 +
To classify data compute pdfgauss and multiply by Prior(w) for each class and choose a class which shows maximum value
 +
 +
 +
'''Generate Multivariate Gaussianl RV''' --jin-young.kim.1, Sun, 17 Feb 2008 15:45:54 -0500
 +
 +
I upload this again since somebody deleted my previous post(Feb/12/2008)
 +
 +
This is an example to generate multivariate Gaussian RV using Matlab. It will generate two classes with the specified mean and covariance.
 +
 +
%% Dimension  = 2
 +
%% Mean and Covariance
 +
mu1 = [1 -1]';
 +
sigma1 = [.9 .4; .4 .3];
 +
mu2 = [0.5 1.5]';
 +
sigma2 = [.9 .4; .4 .3];
 +
mu = [ mu1; mu2];
 +
 +
%% Generate multivariate Gaussian rv
 +
X1 = mvnrnd(mu1,sigma1,N1);    %% N1xdim
 +
X2 = mvnrnd(mu2,sigma2,N2);    %% N2xdim
 +
 +
X1, X2 are 2-dim Gaussian with the correspoinding mean and covariance.
 +
 +
'''Generate Multivariate Gaussianl RV''' --josiah.yoder.1, Tue, 19 Feb 2008 10:50:12 -0500
 +
 +
Sorry, jin-young, I moved your contribution to  [HW: Matlab resources] while cleaning up.  I'll be sure to make a clear link next time.
  
  
−
Question 1: Design and execute an experiment that illustrates the Central Limit Theorem. (You may use problem 5 in DHS p. 80 for inspiration.)
+
'''HW 1 Problem 3 & 4 doubts?''' --ka.ki.ng.1, Sun, 17 Feb 2008 16:54:20 -0500
  
 +
I just a subset of data from one class and do a MLE on it. Then I do classification on the remaining data using the MLE results obtained. I do this class 1 and class 2  separately.
  
  
−
Question 2: Consider n-dimensional feature vectors coming from two classes. Assume that the distributions of the feature vectors for the two classes are (known) normal distributions and that the priors for the classes P(w1) and P(w2) are also known. Write a computer program that classifies the feature vectors according to Bayes decision rule. Generate some artificial (normally distributed) data, and test your program on the data you generated. Try feature vectors of various dimensions. Quantify the accuracy of your results. How does the dimension of the feature vectors affect accuracy? (You may use problem 2 in DHS p. 80 for inspiration.)
+
'''MLE Question''' --neil.d.bedwell.1, Mon, 18 Feb 2008 00:20:08 -0500
  
 +
After obtaining means and covariances from a subset of data from question 2, do we use the prior probabilities to classify the data?
  
 +
In other words, do we know Pw1 and Pw2 for classification using our MLE estimates?
  
−
Question 3: Take a subset of the data you used for Question 2. Use maximum likelihood estimation to estimate the parameters of the feature distribution. Experiment to illustrate the accuracy of the classifier obtained with this estimate. Then repeat the experiments using approximately Gaussian data generated using your answer in Question 1.
+
'''MLE Question''' --josiah.yoder.1, Tue, 19 Feb 2008 10:56:10 -0500
  
 +
You can use MLE to estimate them, too.  But it is reasonable to just assume Pw1 = Pw2.
  
 +
'''MATLAB SCRIPTS''' --wadzanai.d.lunga.1, Wed, 20 Feb 2008 17:37:05 -0500
  
−
Question 4: Replace the words “maximum likelihood estimation” by “Bayesian parameter estimation” in Question 3.
+
Please lets keep the H/W discussions and postings in a more clean representation. If you have a matlab or c script please upload it on the seperate page labelled software rather than posting in on this discussion page. I think that way we can easily search and access the scripts. label what the code does to make it it easy for searching.
  
−
Here is a tutorial on [[Generating Gaussian Samples_OldKiwi]], although the Matlab command normrnd accomplishes this as well.
+
Thank you.

Latest revision as of 10:41, 24 March 2008


Due Tuesday February 19, 2008

Guidelines

Write a short report to present your results. Be sure to include all the relevant graphs as well as a copy of your code. Teamwork is encouraged, but the write up of your report must be your own. Please write the names of your collaborators on the cover page of your report.

Question 1

Design and execute an experiment that illustrates the Central Limit Theorem. (You may use problem 5 in DHS p. 80 for inspiration.)

Question 2

Consider n-dimensional feature vectors coming from two classes. Assume that the distributions of the feature vectors for the two classes are (known) normal distributions and that the priors for the classes P(w1) and P(w2) are also known. Write a computer program that classifies the feature vectors according to Bayes decision rule. Generate some artificial (normally distributed) data, and test your program on the data you generated. Try feature vectors of various dimensions. Quantify the accuracy of your results. How does the dimension of the feature vectors affect accuracy? (You may use problem 2 in DHS p. 80 for inspiration.)

Question 3

Take a subset of the data you used for Question 2. Use maximum likelihood estimation to estimate the parameters of the feature distribution. Experiment to illustrate the accuracy of the classifier obtained with this estimate. Then repeat the experiments using approximately Gaussian data generated using your answer in Question 1.

Question 4

Replace the words “maximum likelihood estimation” by “Bayesian parameter estimation” in Question 3.

Here is a tutorial on Generating Gaussian Samples_OldKiwi, although the Matlab command normrnd accomplishes this as well.


Discussions

Illustrating the Central Limit Theorem and dice-rolling experiment -- jungtag.gong.1

- `Illustrating the central limit theorem <http://www.stat.wvu.edu/SRS/Modules/CLT/clt.html>`_ : This web site gives basic principles, illustrating the centeral limit theorem when the underlying distribution is normal or chi-square or uniform, bowtie, right wedge, left wedge, and triangular and exercises.

- `Central limit theorem demo applet using simulated dice-rolling experiments <http://www.stat.sc.edu/~west/javahtml/CLT.html>`_  : This applet demonstrates the central limit theorem using simulated dice-rolling experiments.

- `Central Limit Theorem explanation and applet <http://www.math.csusb.edu/faculty/stanton/probstat/clt.html>`_ : introducing the central limit theorem and showing applet to demonstrate the central limit theorem using simulated dice rolloing experiments

It's a method to generate test data with multivariate normal distribution using randn, sqrtm, and repmat in Matlab** -- jungtag.gong.1

RANDN(N,Dim) * SQRTM(covar) + REPMAT(mean, N, 1);

, where N is the number of data, Dim is the dimension of feature vecotrs.

- RANDN : Normally distributed random numbers : ex) RANDN(N) is an N-by-N matrix with random entries, chosen from a normal distribution with mean zero, variance one and standard deviation one.

- SQRTM : Matrix square root : ex) X = SQRTM(A) is the principal square root of the matrix A, i.e. X*X = A.

- REPMAT : Replicate and tile an array : ex) B = repmat(A,M,N) creates a large matrix B consisting of an M-by-N tiling of copies of A.


Central Limit Theorem - Galton's Experiment --jin-young.kim.1, Fri, 08 Feb 2008 23:50:57 -0500

1. The Central Limit Theorem says that for large values of the parameter n, the binomial distribution is approximately normal. Galton used the quincunx to show this.

Here is a interesting video to show the experiment http://inter.heao.han.nl/opleidingen/rstnet/rst2/Applets/galton%20board/galtonintro.htm

2. Application of CLT to signal processing Signals can be smoothed by applying a Gaussian filter, which is just the convolution of a signal with an appropriately scaled Gaussian function. Due to the central limit theorem this smoothing can be approximated by several filter steps that can be computed much faster, like the simple moving average.

Ref: "http://en.wikipedia.org/wiki/Central_limit_theorem"

A question regarding Homework:

When we initialise Random Vectors How do we do so for a given covariance matrix ? Will it be good enough if I use the randn function ro generate a vecort of n-gaussian random variables ? can this be called a random vector ? BAsically my question is how do we simulate gaussian data whcih has a NON ZERO elemts in places OTHER than the main diagonal ?


The Central Limit Theorem vs. The Law of Large Number --jin-young.kim.1, Sun, 10 Feb 2008 20:20:11 -0500

=> The Central Limit Theorem contains more information than the Law of Large Numbers, because it gives us detailed information about the shape of the distribution of Sn.

=> For large n the shape is approximately the same as the shape of the standard normal density. More specically, the Central Limit Theorem says that if we standardize and height-correct the distribution of Sn, then the normal density function is a very good approximation to this distribution when n is large.

=> Thus, we have a computable approximation for the distribution for Sn, which provides us with a powerful technique for generating answers for all sorts of questions about sums of independent random variables, even if the individual random variables have different distributions.

Ref: "Intorduction to Probability," by Charles M. Grinstead and J. Laurie Snell


Central Limit Theorem's link (includes a nice applet) --marc.bosch.ruiz.1, Sun, 10 Feb 2008 21:18:12 -0500

`Central Limit Theorem applet <http://www.math.csusb.edu/faculty/stanton/probstat/clt.html>`_


Useful Matlab Functions --stephen.r.rudolph.1, Sun, 10 Feb 2008 21:38:45 -0500

The function "normrnd" will generate an array of random, normal ([Gaussian distribution]) values. To use normrnd, you can pass it 3 parameters. The first parameter is the mean, the second is the [standard deviation], and the third is a tuple specifying the array dimensions. Here is an example for a 10 by 1 matrix of random normal values based on a mean of 100 and a standard deviation of 20:


normrnd(100, 20, [10 1])

ans =

105.8882 73.2764 114.2865 132.4712 86.1645 117.1599 125.0800 68.1254 71.1807 111.4230


Sampling from Multivariate Normal --ashish.kamra.1, Sun, 10 Feb 2008 22:59:46 -0500

For those looking for a routine to sample points from a multivariate normal density, this library of C functions may come in handy ..

http://www.netlib.org/random/ranlib.c.tar.gz


Hw 1 Problem 2 doubt ? --ashish.kamra.1, Fri, 15 Feb 2008 09:04:42 -0500

How are the test samples supposed to be generated ? If I generate them using the distribution from class 1, then they will classified as class by the bayes classifier unless I choose the distributions of both the classes very close to each other. So essentially the classification accuracy will depend upon how well separated the classes are ? Is that correct ? Or am I missing something here ...

Hw 1 Problem 2 doubt ? --josiah.yoder.1, Sat, 16 Feb 2008 12:35:08 -0500

Yes. That is correct.


HW 1 Problem 3 & 4 doubts? --yamini.nimmagadda.1, Sun, 17 Feb 2008 13:03:32 -0500

    • problem 3**: How can we estimate parameters using MLE from the data generated using approximate Gaussian distribution (from question 1) for 2 categories ?? In question 1 we get a single distribution which can have only one set of parameters that belong to one class. But it is required to estimate parameters for 2 classes in question 3.

Re) HW 1 Problem 3 & 4 doubts? --jin-young.kim.1, Sun, 17 Feb 2008 14:35:42 -0500

I think we just need to separately estimate parameters for each class. Finding parameters is done in training stage, so I think we have alreay known to which classes the data belongs.


problem 4: For the BPE, what should we assume for the parameters of the distribution of mean? What about the variance?


HW1 - 2) How to easily compute p(x/w)*Prior(w) using Matlab --jin-young.kim.1, Sun, 17 Feb 2008 14:26:48 -0500

To classify data using Bayesian classifier we already know Prior(w) and need to compute p(x/w). When p is multidimensioanl Gaussian, we can use Matlab internal function "mvnpdf".

Ex) mvnpdf(X,Mean,Cov)

X <= data we want to classify Mean <= already known when created Cov <= already known when created

To classify data compute pdfgauss and multiply by Prior(w) for each class and choose a class which shows maximum value


Generate Multivariate Gaussianl RV --jin-young.kim.1, Sun, 17 Feb 2008 15:45:54 -0500

I upload this again since somebody deleted my previous post(Feb/12/2008)

This is an example to generate multivariate Gaussian RV using Matlab. It will generate two classes with the specified mean and covariance.

%% Dimension = 2 %% Mean and Covariance mu1 = [1 -1]'; sigma1 = [.9 .4; .4 .3]; mu2 = [0.5 1.5]'; sigma2 = [.9 .4; .4 .3]; mu = [ mu1; mu2];

%% Generate multivariate Gaussian rv X1 = mvnrnd(mu1,sigma1,N1);  %% N1xdim X2 = mvnrnd(mu2,sigma2,N2);  %% N2xdim

X1, X2 are 2-dim Gaussian with the correspoinding mean and covariance.

Generate Multivariate Gaussianl RV --josiah.yoder.1, Tue, 19 Feb 2008 10:50:12 -0500

Sorry, jin-young, I moved your contribution to [HW: Matlab resources] while cleaning up. I'll be sure to make a clear link next time.


HW 1 Problem 3 & 4 doubts? --ka.ki.ng.1, Sun, 17 Feb 2008 16:54:20 -0500

I just a subset of data from one class and do a MLE on it. Then I do classification on the remaining data using the MLE results obtained. I do this class 1 and class 2 separately.


MLE Question --neil.d.bedwell.1, Mon, 18 Feb 2008 00:20:08 -0500

After obtaining means and covariances from a subset of data from question 2, do we use the prior probabilities to classify the data?

In other words, do we know Pw1 and Pw2 for classification using our MLE estimates?

MLE Question --josiah.yoder.1, Tue, 19 Feb 2008 10:56:10 -0500

You can use MLE to estimate them, too. But it is reasonable to just assume Pw1 = Pw2.

MATLAB SCRIPTS --wadzanai.d.lunga.1, Wed, 20 Feb 2008 17:37:05 -0500

Please lets keep the H/W discussions and postings in a more clean representation. If you have a matlab or c script please upload it on the seperate page labelled software rather than posting in on this discussion page. I think that way we can easily search and access the scripts. label what the code does to make it it easy for searching.

Thank you.

Retrieved from "https://www.projectrhea.org/rhea/index.php?title=ECE662:Homework_1_OldKiwi&oldid=27669"
Shortcuts
Help Main Wiki Page Random Page Special Pages Log in
Search

Alumni Liaison

Basic linear algebra uncovers and clarifies very important geometry and algebra.

Dr. Paul Garrett