Bayersian Parameter Estimation: Gaussian Case - Rhea

Introduction: Bayesian Estimation

Although the estimator obtained from Maximum Likelihood Estimation (MLE) and Bayersian Parameter Estimation(BPE) would be similar or even identical for most of the time, the key idea(structure) for MLE and BPE is completely different. For Maximum Likelihood Estimation, we can consider the parameter estimated to be a fixed number (or several numbers if more than one parameters), while in BPE the estimated parameter is a vector (r.v.).

To start with, Bayes' formula was transformed into the following form given samples class $ \mathcal{D} $:

Based on the observation on above equations, it can be concluded that both class-conditional densities and the priori could be obtained based on the training data.

Now, assuming that the we are working on a supervised case with labelled training data, that is all samples from the training data could be separated accurately into c subsets $ \mathcal{D}_1, \mathcal{D}_2, ..., \mathcal{D}_c $.

Hence, the above equation could be further developed into the following form:

Now, assume that a set of $ N $ independent samples were obtained from a certain class $ \mathcal{D} = \{x_1, x_2, ... , x_N \} $ and for each of the sample there exist a probability function with the parameter form: p(x). In order to form a BPE estimation, we will consider $ \theta $ to be a vector (random variable). More specifically, a probability function given a class condition of D and a parameter vector of $ \theta $ is defined as below:

$ p(x|D) $ can be computed as:

Bayesian Parameter Estimation: General Theory

In order to provide better understanding regarding Bayesian Parameter Estimation (BPE) technique, first of all we will briefly discuss the general technique. For the BPE method, as $ \theta $ is considered to be a random variable (vector) hence it is assumed to be unknown. Although $ \theta $ in general is unknown, another assumption need to be made that $ \theta $ has the priori distribution of the form $ p(\theta) $ which is considered to be known. Hence, in order to estimate the parameter $ \theta $ both the information in priori and the information from set $ \mathcal{D} $ of n samples $ x_1, x_2, ... , x_n $ need to be utilized. Since the training data is known and well labelled, obviously the density function of a sample x with parameter $ \theta $ is known, denoted as $ p(x|\theta) $.

From the previous section we have already obtained:

Furthermore, by Bayes Theorem (with some transformation),

Although we are very close already, we still need to substitute the class condition 'D' with the samples $ x_k $, based on our assumption made at the beginning of this section. In order to do that, first the probability function of class 'D' with $ \theta $ as a parameter need to be computed:

Now for a class $ \mathcal{D} $ which contains 'N' samples: $ \mathcal{D}^n = \{x_1, x_2, ... x_n\} $, we can further transform the above equation to the following form, as $ p(x_n|\theta) $ is assumed to be known and $ p(D^{n-1}|\theta) $ is the probability function of class D with N-1 samples:

Hence, after we substituting class condition 'D' with samples $ x_k $, the Bayesian Parameter Estimation equation then transformed into the following form:

Bayesian Parameter Estimation: Gaussian Case

The Univariate Case: $ p(\mu|\mathcal{D}) $

As was done in MLE first will start with a simple case with only the mean: $ \mu $ unknown. As usual we will assume sample $ x_k $ is normally distributed as:

and the parameter $ \mu $ has the distribution of:

as parameter $ \mu $ is not estimated to be a number but a random variable.

Using Bayes' formula and the corresponding derivation from the previous section the corresponding function could be easily obatined:

Where $ \alpha $ is introduced as a 'scale' coefficient in order to simplify the derivation. Please note that $ \alpha $ is completely independent of $ \mu $.

As $ x_k $ is normally distributed we update the $ p(x_k|\mu) $ and $ p(u) $ with the known distribution function:

Again, substitute $ p(x_k|\mu) $ and $ p(u) $ in equation $ p(\mu|D) = \alpha \prod_{k = 1}^n p(x_k|\mu)p(\mu) $, we obtained:

Similarly, in order to simplify the derivation we update the scaling factors to $ \alpha' $ and $ \alpha'' $, and correspondingly,

Finally, compare derived $ p(u|D) $ to the Gaussian Distribution in the standard form:

Based on knowledge on Gaussian Distribution, $ \mu_n $ and $ \sigma_n^2 $ could be obtained accordingly:

(Why is it important to emphasize both variance: $ \sigma_n^2 $ and mean: $ \mu_n $? Since the distribution is in Gaussian Form and the two most decisive parameters in Gaussian R.V. are mean and variance) Please note that $ \bar{x_n} $ is the empirical mean in our known training data.

The Univariate Case: $ p(x|\mathcal{D}) $

Given the posteriori density $ p(\mu|D) $ successfully derived (variance: $ \sigma_n^2 $ and mean: $ \mu_n $ now known), the final step is to estimate $ p(x|D) $ based on the conclusions in previous sections.

Based on the text Duda's chatpter #3.4.2 and Prof. Mimi's notes:

Finally, substitute $ \sigma_n^2 $ and $ \mu_n $ the probability function $ p(x|\mathcal{D}) $ is obtained:

Hence, $ p(x|D) $ is normally distributed as:

References

[1]. Mireille Boutin, "ECE662: Statistical Pattern Recognition and Decision Making Processes," Purdue University, Spring 2014.

[2]. R. Duda, P. Hart, Pattern Classification. Wiley-Interscience. Second Edition, 2000.

Questions and comments

If you have any questions, comments, etc. please post them on this page.

Back to ECE 662 S14 course wiki

Back to ECE 662 course page