ECE662Selecture zhenpengMLE - Rhea

1. Motivation

• Most likely converge as number of number of training sample increase.
• Simpler than alternate methods such as Bayesian technique.

2. MLE as a Parametric Density Estimation

• Statistical Density Theory Context
  • Given c classes + some knowledge about features $ x \in \mathbb{R}^n $ (or some other space)
  • Given training data, $ x_j\sim\rho(x)=\sum\limits_{i=1}^n\rho(x|w_i) Prob(w_i) $, unknown class $ w_{ij} $ for $ x_j $ is know, $ \forall{j}=1,...,N $ (N hopefully large enough)
  • In order to make decision, we need to estimate $ \rho(x|w_i) $, $ Prob(w_i) $ $ \rightarrow $ use Bayes rule, or $ \rho(x|w_i) $ $ \rightarrow $ use Neyman-Pearson Criterion
  • To estimate the above two, use training data.

• The parametric pdf|Prob estimation problem
  • Let $ D={x_1,x_2,...,x_N} $, $ x_j $ is drown independently from some probability law.
  • Choose parametric from $ \rho(x|\theta) $ for the pdf of x or $ Prob(x|\theta) $ for the probability of x $ \rightarrow $ an unknown parametric vector
  • Use $ D $ to estimate $ \theta $

• Definition: The maximum likelihood estimate of $ \theta $ is the value $ \hat{\theta} $ that maximize $ \rho_D(D|\theta) $, if x is continuous R.V., or $ Prob(D|\theta) $, if x is discrete R.V.

• Observation: By independence, $ \rho(D|\theta)=\rho(x_1,x_2,...,x_N|\theta) $ = $ \prod\limits_{j=1}^n\rho(x_j|\theta) $
  • Simple Example One:

Those to estimate the priors: $ Prob(w_1), Prob(w_2) $ for $ c=2 $ classes.

Let $ Prob(w_1)=P $, $ \Rightarrow $ $ Prob(w_2)=1-P $, as an unknown parameter ($ \theta=P $)

Let $ w_j $ be the class of some $ x_j $, ($ j\in{1,2,...N} $)

$ Prob(D|P) $ = $ \prod\limits_{j=1}^n Prob(w_{ij}|P) $, $ x\sim \rho(x) $

=$ \prod\limits_{j=1}^{N_1} Prob(w_{ij}|P)\prod\limits_{j=1}^{N_2}Prob(w_{ij}|p) $

=$ P^{N_1}\dot(1-P)^{N-N_1} $

, the first $ w_{ij}=w_1 $ and the second $ w_{ij}=w_2 $,

$ N1 $= number of sample from class 1 Then, we $ \infty $ differentiate P $ (Prob(D|P)) $, so local max is where derivative = 0.

$ \frac{d}{dP} Prob(D|P)=\frac{d}{dP} P^{N_1}(1-P)^{N-N_1} $

$ =N_1P^{N_1-1}(1-P)^{N-N_1}-(N-N_1)P^{N_1}(1-p) $

$ =p^{N_1-1}(1-P)^{N-N_1-1}[N_1(1-P)-(N-N_1)P]=0 $

$ \Rightarrow $ So either P=0 or P=1 $ \rightarrow N_1(1-P) $

$ \Leftrightarrow P=\frac{N_1}{N} $

• • Simple Example Two: Continuous R.V.: Estimate mean of Gaussian with Known $ \Sigma $

$ \rho(\vec{x}|\vec{\mu})=N(\vec{\mu},\Sigma) $, where $ \mu $ is

unknown, and $ Sigma $ is known. 

$ \rho(D|\vec{\mu}) = \prod\limits_{j=1}^{N}\rho(x_j|\vec{\mu}) $

Observe the MLE $ \in \hat{\theta} $, also maximize $ log\rho_D(D|\theta) $ since log is monotonic

= $ \sum\limits_{j=1}^{N}ln(\frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}}) $ $ \exp^{-\frac{(x_j-\vec{\mu})^T\Sigma^{-1}(x_j-\vec{\mu})}{2}} $

= $ \sum\limits_{j=1}^{N}ln(\frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}}) $ $ -\frac{(x_j-\vec{\mu})^T\Sigma^{-1}(x_j-\vec{\mu})}{2} $

which is $ \infty $ many times differentiable for $ \vec{\mu} $, so local max are where $ \nabla=0 $

compute $ \nabla $, $ \nabla_{\vec{\mu}}ln\rho_{D}(D|\vec{\mu}) $

=$ \sum\limits_{j=1}^{N}\nabla_{\vec{\mu}} (ln(\frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}}) $ $ -\frac{(x_j-\vec{\mu})^T\Sigma^{-1}(x_j-\vec{\mu})}{2}) $

=$ -1/2\sum\limits_{j=1}^{N}\nabla_{\vec{\mu}}[(x_j-\vec{\mu})^T\Sigma^{-1}(x_j-\vec{\mu})] $

=$ -1/2\sum\limits_{j=1}^{N} \begin{bmatrix} \frac{\partial}{\partial\mu_1} (x_j-{\mu})^T\Sigma^{-1}(x_j-{\mu})\\ \frac{\partial}{\partial\mu_2} (x_j-{\mu})^T\Sigma^{-1}(x_j-{\mu})\\ \vdots \\ \frac{\partial}{\partial\mu_n} (x_j-{\mu})^T\Sigma^{-1}(x_j-{\mu})\\ \end{bmatrix} $

But $ \frac{\partial}{\partial\mu_1} (x_j-{\mu})^T\Sigma^{-1}(x_j-{\mu}) $

=$ (\frac{\partial}{\partial \mu_i}(x_j-\mu)^T)\Sigma^{-1} $ $ (x_j-\mu)+(x_j-\mu)^T\Sigma^{-1}\frac{\partial}{\partial \mu_i}(x_j-\mu) $

=$ 2\frac{\partial}{\partial \mu_i}(x_j-\mu)^T)\Sigma^{-1}(x_j-\mu) $

=$ 2(0,0,0,...,-1,0,...,0)\Sigma^{-1}(x_j-\mu) $

=$ -2\vec{e_i}^{T}\Sigma^{-1}(x_j-\mu) $

so, $ \nabla{ln\rho_D(D|\mu)} = -1/2\sum\limits_{j=1}^{N} $ $ \begin{bmatrix} -2\vec{e_1}^{T}\Sigma^{-1}(x_j-{\mu})\\ -2\vec{e_2}^{T}\Sigma^{-1}(x_j-{\mu})\\ \vdots \\ -2\vec{e_n}^{T}\Sigma^{-1}(x_j-{\mu})\\ \end{bmatrix} $

=$ \sum\limits_{j=1}^{N} $ $ \begin{bmatrix} -2\vec{e_1}^{T}\\ -2\vec{e_2}^{T}\\ \vdots \\ -2\vec{e_n}^{T}\\ \end{bmatrix} $ $ \Sigma^{-1}(x_j-\mu) $, the vector of $ \vec{e_i} $ is the space domain of feature

=$ \sum\limits_{j=1}^{N}\Sigma^{-1}(x_j-\mu) $

=$ \Sigma^{-1}\sum\limits_{j=1}^{N}(x_j-\mu) $ set to be 0

$ \Rightarrow \Sigma\Sigma^{-1}\sum\limits_{j=1}^{N}\Sigma^{-1}(x_j-\mu) = \Sigma \cdot 0 $

$ \Rightarrow \sum\limits_{j=1}^{N}(x_j-\mu) = 0 $

$ \Rightarrow \frac{1}{N}\sum\limits_{j=1}^{N}x_j = \mu $

$ \rightarrow $ the sample mean is the maximum likelihood estimate for $ \mu $

(create a question page and put a link below)

Questions and comments

If you have any questions, comments, etc. please post them on https://kiwi.ecn.purdue.edu/rhea/index.php/ECE662Selecture_ZHenpengMLE_Ques.

Back to ECE662, Spring 2014