2.5 Central limit theorem
From the course notes on "sequence of random variables" of Sangchun Han, ECE PhD student.
Let $ \left\{ \mathbf{X}_{n}\right\} $ be a sequence of i.i.d. random vectors with mean μ and variance σ2 , such that 0 < σ2 < ∞ . Then if
$ \mathbf{Z}_{n}\triangleq\frac{\left(\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n}\right)-n\mu}{\sigma\sqrt{n}} $
then $ \mathbf{Z}_{n} $ converges in distribution to a random variable $ \mathbf{Z} $ that is Gaussian with mean 0 and variance 1 .
In fact, $ \mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n} $ has the mean nμ and the variance nσ2 .
i.e. $ F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx $ as $ n\rightarrow\infty , \forall z\in\mathbf{R} $ .
Proof
We will show that $ \Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow\left(n\rightarrow\infty\right)\longrightarrow e^{-\frac{1}{2}\omega^{2}} , \forall\omega\in\mathbf{R} $ .
$ \Phi_{\mathbf{Z}_{n}}\left(\omega\right)=E\left[e^{i\omega\mathbf{Z}}\right]=E\left[\exp\left\{ \frac{i\omega}{\sigma\sqrt{n}}\sum_{k=1}^{n}\left(\mathbf{X}_{k}-\mu\right)\right\} \right]=E\left[\prod_{k=1}^{n}e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right] $$ =\prod_{k=1}^{n}E\left[e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right]=\cdots=\left(E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]\right)^{n}. $
We can expand the exponential as a power series (in ω about ω = 0 ).
$ E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right] $
It can be shown that $ \frac{E\left[R\left(\omega\right)\right]}{\omega^{2}/2n}\longrightarrow0 $ as $ n\rightarrow\infty , \forall\omega\in\mathbf{R} $ .
Thus we have
$ E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]=1-\frac{\omega^{2}}{2n}+E\left[R\left(\omega\right)\right]\backsimeq1-\frac{\omega^{2}}{2n}\text{ as }n\rightarrow\infty. $
$ \Phi_{\mathbf{Z}_{n}}\left(\omega\right)\backsimeq\left(1-\frac{\omega^{2}}{2n}\right)^{n}=\left(1+\frac{\left(-\omega^{2}/2\right)}{n}\right)^{n}\longrightarrow e^{-\omega^{2}/2}\text{ as }n\rightarrow\infty. $
$ \therefore\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow e^{-\omega^{2}/2}\Longrightarrow F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx. $
We have “proved” the central limit theorem for i.i.d. random variables. It can be proven under much more general circumstances (e.g. sums of correlated sequences of random variables). Lindeberg-Feller central limit theorem works for some classes of correlated random variables.
Gaussian normalization
Gaussian distribution $ \left(\mathbf{X}\right) $ with μ and σ2⇒ Standard normal distribution $ \left(\mathbf{Z}\right) $ that has mean 0 and variance 1
$ \mathbf{Z}=\frac{\mathbf{X}-\mu}{\sigma}\text{ and }\mathbf{X}=\sigma\mathbf{Z}+\mu. $
