Revision as of 03:53, 5 March 2013 by Mboutin (Talk | contribs)


Examples of Parameter Estimation based on Maximum Likelihood (MLE): the exponential distribution and the geometric distribution

for ECE662: Decision Theory


Exponential Distribution

Let $ {X}_{1}, {X}_{2}, {X}_{3}.....{X}_{n} $ be a random sample from the exponential distribution with p.d.f.

$ f(x;\theta)=\frac{1}{\theta}{e}^{\frac{-x}{\theta}} 0<x<\infty, \theta\in\Omega=\{\theta|0<\theta<\infty\} $

The likelihood function is given by:

$ L(\theta)=L\left(\theta;{x}_{1},{x}_{2}...{x}_{n} \right)=\left(\frac{1}{\theta}{e}^{\frac{{-x}_{1}}{\theta}}\right)\left(\frac{1}{\theta}{e}^{\frac{{-x}_{2}}{\theta}}\right)...\left(\frac{1}{\theta}{e}^{\frac{{-x}_{n}}{\theta}} \right)=\frac{1}{{\theta}^{n}}exp\left(\frac{-\sum_{1}^{n}{x}_{i}}{\theta} \right) $

Taking log, we get,

$ lnL\left(\theta\right)=-\left(n \right)ln\left(\theta\right) -\frac{1}{\theta}\sum_{1}^{n}{x}_{i}, 0<\theta<\infty $

Differentiating the above expression, and equating to zero, we get

$ \frac{d\left[lnL\left(\theta\right) \right]}{d\theta}=\frac{-\left(n \right)}{\left(\theta\right)} +\frac{1}{{\theta}^{2}}\sum_{1}^{n}{x}_{i}=0 $

The solution of equation for $ \theta $ is:

$ \theta=\frac{\sum_{1}^{n}{x}_{i}}{n} $

Thus, the maximum likelihood estimator of $ \Theta $ is

$ \Theta=\frac{\sum_{1}^{n}{X}_{i}}{n} $


Geometric Distribution

Let $ {X}_{1}, {X}_{2}, {X}_{3}.....{X}_{n} $ be a random sample from the geometric distribution with p.d.f.

$ f\left(x;p \right)={\left(1-p \right)}^{x-1}p, x=1,2,3.... $

The likelihood function is given by:

$ L\left(p \right)={\left(1-p \right)}^{{x}_{1}-1}p {\left(1-p \right)}^{{x}_{2}-1}p...{\left(1-p \right)}^{{x}_{n}-1}p ={p}^{n}{\left(1-p \right)}^{\sum_{1}^{n}{x}_{i}-n} $

Taking log,

$ lnL\left(p \right)= nln{p}+\left(\sum_{1}^{n}{x}_{i}-n \right)ln{\left(1-p \right)} $

Differentiating and equating to zero, we get,

$ \frac{d\left[lnL\left(p \right)\right]}{dp}=\frac{n}{p} -\frac{\left(\sum_{1}^{n}{x}_{i}-n \right)}{\left(1-p \right)}=0 $

Therefore,

$ p=\frac{n}{\left(\sum_{1}^{n}{x}_{i} \right)} $

So, the maximum likelihood estimator of P is:

$ P=\frac{n}{\left(\sum_{1}^{n}{X}_{i} \right)}=\frac{1}{X} $

This agrees with the intuition because, in n observations of a geometric random variable, there are n successes in the $ \sum_{1}^{n}{X}_{i} $ trials. Thus the estimate of p is the number of successes divided by the total number of trials.


More examples: Binomial and Poisson Distributions

Back to Lecture 7: "Comparison of Maximum likelihood (MLE) and Bayesian Parameter Estimation"

Back to ECE662, Spring 2008, Prof. Boutin

Alumni Liaison

Abstract algebra continues the conceptual developments of linear algebra, on an even grander scale.

Dr. Paul Garrett