1.12 Minimum Mean-Square Error Estimation
From the ECE600 Pre-requisites notes of Sangchun Han, ECE PhD student.
Let $ \mathbf{X} $ and $ \mathbf{Y} $ be two jointly-distributed RVs, suppose we want to estimate the value of $ \mathbf{Y} $ given the value of $ \mathbf{X} $ (i.e. given that we observe$ \left\{ \mathbf{X}=x\right\} $ ). What is the “best” estimate of $ \mathbf{Y} $ ? One commonly used error criterion is square-error. The goal then becomes to minimize the mean-square error. We wish to find a function $ c\left(x\right) $ to estimate $ \mathbf{Y} $ given that $ \mathbf{X}=x $ such that $ \epsilon=E\left[\left(\mathbf{Y}-c\left(\mathbf{X}\right)\right)^{2}\right] $ is minimized.
Claim
The mean-square error is minimized by the function $ c\left(x\right)=E\left[\mathbf{Y}|\mathbf{X}=x\right] $.
We will use the following notation.
$ \hat{y}_{MMS}\left(x\right)=E\left[\mathbf{Y}|\mathbf{X}=x\right] $
$ \hat{x}_{MMS}\left(y\right)=E\left[\mathbf{X}|\mathbf{Y}=y\right] $
Maximum Aposteriori Probability estimator
$ \hat{y}_{MAP}\left(x\right)=\arg\max_{y}\left\{ f_{\mathbf{Y}}\left(y|x\right)\right\} $
$ \hat{x}_{MAP}\left(y\right)=\arg\max_{x}\left\{ f_{\mathbf{X}}\left(x|y\right)\right\} $
