| Line 6: | Line 6: | ||
From the [[ECE_600_Sequences_of_Random_Variables|course notes on "sequence of random variables"]] of [[user:han84|Sangchun Han]], [[ECE]] PhD student. | From the [[ECE_600_Sequences_of_Random_Variables|course notes on "sequence of random variables"]] of [[user:han84|Sangchun Han]], [[ECE]] PhD student. | ||
---- | ---- | ||
| − | Let <math>\left\{ \mathbf{X}_{n}\right\}</math> be a sequence of <span class="texhtml">''i''.''i''.''d''.</span> random vectors with mean <span class="texhtml">μ</span> and variance <span class="texhtml">σ<sup>2</sup></span> , such that <span class="texhtml">0 < σ<sup>2</sup> < ∞</span> . Then if | + | Let <math class="inline">\left\{ \mathbf{X}_{n}\right\}</math> be a sequence of <span class="texhtml">''i''.''i''.''d''.</span> random vectors with mean <span class="texhtml">μ</span> and variance <span class="texhtml">σ<sup>2</sup></span> , such that <span class="texhtml">0 < σ<sup>2</sup> < ∞</span> . Then if |
| − | <math>\mathbf{Z}_{n}\triangleq\frac{\left(\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n}\right)-n\mu}{\sigma\sqrt{n}}</math> | + | <math class="inline">\mathbf{Z}_{n}\triangleq\frac{\left(\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n}\right)-n\mu}{\sigma\sqrt{n}}</math> |
| − | then <math>\mathbf{Z}_{n}</math> converges in distribution to a random variable <math>\mathbf{Z}</math> that is Gaussian with mean 0 and variance 1 . | + | then <math class="inline">\mathbf{Z}_{n}</math> converges in distribution to a random variable <math class="inline">\mathbf{Z}</math> that is Gaussian with mean 0 and variance 1 . |
| − | In fact, <math>\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n}</math> has the mean <span class="texhtml">''n''μ</span> and the variance <span class="texhtml">''n''σ<sup>2</sup></span> . | + | In fact, <math class="inline">\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n}</math> has the mean <span class="texhtml">''n''μ</span> and the variance <span class="texhtml">''n''σ<sup>2</sup></span> . |
| − | i.e. <math>F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx</math> as <math>n\rightarrow\infty , \forall z\in\mathbf{R}</math> . | + | i.e. <math class="inline">F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx</math> as <math class="inline">n\rightarrow\infty , \forall z\in\mathbf{R}</math> . |
Proof | Proof | ||
| − | We will show that <math>\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow\left(n\rightarrow\infty\right)\longrightarrow e^{-\frac{1}{2}\omega^{2}} , \forall\omega\in\mathbf{R}</math> . | + | We will show that <math class="inline">\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow\left(n\rightarrow\infty\right)\longrightarrow e^{-\frac{1}{2}\omega^{2}} , \forall\omega\in\mathbf{R}</math> . |
| − | <math>\Phi_{\mathbf{Z}_{n}}\left(\omega\right)=E\left[e^{i\omega\mathbf{Z}}\right]=E\left[\exp\left\{ \frac{i\omega}{\sigma\sqrt{n}}\sum_{k=1}^{n}\left(\mathbf{X}_{k}-\mu\right)\right\} \right]=E\left[\prod_{k=1}^{n}e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right]</math><math>=\prod_{k=1}^{n}E\left[e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right]=\cdots=\left(E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]\right)^{n}.</math> | + | <math class="inline">\Phi_{\mathbf{Z}_{n}}\left(\omega\right)=E\left[e^{i\omega\mathbf{Z}}\right]=E\left[\exp\left\{ \frac{i\omega}{\sigma\sqrt{n}}\sum_{k=1}^{n}\left(\mathbf{X}_{k}-\mu\right)\right\} \right]=E\left[\prod_{k=1}^{n}e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right]</math><math class="inline">=\prod_{k=1}^{n}E\left[e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right]=\cdots=\left(E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]\right)^{n}.</math> |
We can expand the exponential as a power series (in <span class="texhtml">ω</span> about <span class="texhtml">ω = 0</span> ). | We can expand the exponential as a power series (in <span class="texhtml">ω</span> about <span class="texhtml">ω = 0</span> ). | ||
| − | <math>E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]</math> | + | <math class="inline">E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]</math> |
| − | It can be shown that <math>\frac{E\left[R\left(\omega\right)\right]}{\omega^{2}/2n}\longrightarrow0</math> as <math>n\rightarrow\infty , \forall\omega\in\mathbf{R}</math> . | + | It can be shown that <math class="inline">\frac{E\left[R\left(\omega\right)\right]}{\omega^{2}/2n}\longrightarrow0</math> as <math class="inline">n\rightarrow\infty , \forall\omega\in\mathbf{R}</math> . |
Thus we have | Thus we have | ||
| − | <math>E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]=1-\frac{\omega^{2}}{2n}+E\left[R\left(\omega\right)\right]\backsimeq1-\frac{\omega^{2}}{2n}\text{ as }n\rightarrow\infty.</math> | + | <math class="inline">E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]=1-\frac{\omega^{2}}{2n}+E\left[R\left(\omega\right)\right]\backsimeq1-\frac{\omega^{2}}{2n}\text{ as }n\rightarrow\infty.</math> |
| − | <math>\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\backsimeq\left(1-\frac{\omega^{2}}{2n}\right)^{n}=\left(1+\frac{\left(-\omega^{2}/2\right)}{n}\right)^{n}\longrightarrow e^{-\omega^{2}/2}\text{ as }n\rightarrow\infty.</math> | + | <math class="inline">\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\backsimeq\left(1-\frac{\omega^{2}}{2n}\right)^{n}=\left(1+\frac{\left(-\omega^{2}/2\right)}{n}\right)^{n}\longrightarrow e^{-\omega^{2}/2}\text{ as }n\rightarrow\infty.</math> |
| − | <math>\therefore\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow e^{-\omega^{2}/2}\Longrightarrow F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx.</math> | + | <math class="inline">\therefore\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow e^{-\omega^{2}/2}\Longrightarrow F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx.</math> |
We have “proved” the central limit theorem for <span class="texhtml">''i''.''i''.''d''.</span> random variables. It can be proven under much more general circumstances (e.g. sums of correlated sequences of random variables). Lindeberg-Feller central limit theorem works for some classes of correlated random variables. | We have “proved” the central limit theorem for <span class="texhtml">''i''.''i''.''d''.</span> random variables. It can be proven under much more general circumstances (e.g. sums of correlated sequences of random variables). Lindeberg-Feller central limit theorem works for some classes of correlated random variables. | ||
| Line 40: | Line 40: | ||
Gaussian normalization | Gaussian normalization | ||
| − | Gaussian distribution <math>\left(\mathbf{X}\right)</math> with <span class="texhtml">μ</span> and <span class="texhtml">σ<sup>2</sup>⇒</span> Standard normal distribution <math>\left(\mathbf{Z}\right)</math> that has mean 0 and variance 1 | + | Gaussian distribution <math class="inline">\left(\mathbf{X}\right)</math> with <span class="texhtml">μ</span> and <span class="texhtml">σ<sup>2</sup>⇒</span> Standard normal distribution <math class="inline">\left(\mathbf{Z}\right)</math> that has mean 0 and variance 1 |
| − | <math>\mathbf{Z}=\frac{\mathbf{X}-\mu}{\sigma}\text{ and }\mathbf{X}=\sigma\mathbf{Z}+\mu.</math> | + | <math class="inline">\mathbf{Z}=\frac{\mathbf{X}-\mu}{\sigma}\text{ and }\mathbf{X}=\sigma\mathbf{Z}+\mu.</math> |
---- | ---- | ||
Latest revision as of 10:42, 30 November 2010
2.5 Central limit theorem
From the course notes on "sequence of random variables" of Sangchun Han, ECE PhD student.
Let $ \left\{ \mathbf{X}_{n}\right\} $ be a sequence of i.i.d. random vectors with mean μ and variance σ2 , such that 0 < σ2 < ∞ . Then if
$ \mathbf{Z}_{n}\triangleq\frac{\left(\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n}\right)-n\mu}{\sigma\sqrt{n}} $
then $ \mathbf{Z}_{n} $ converges in distribution to a random variable $ \mathbf{Z} $ that is Gaussian with mean 0 and variance 1 .
In fact, $ \mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{n} $ has the mean nμ and the variance nσ2 .
i.e. $ F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx $ as $ n\rightarrow\infty , \forall z\in\mathbf{R} $ .
Proof
We will show that $ \Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow\left(n\rightarrow\infty\right)\longrightarrow e^{-\frac{1}{2}\omega^{2}} , \forall\omega\in\mathbf{R} $ .
$ \Phi_{\mathbf{Z}_{n}}\left(\omega\right)=E\left[e^{i\omega\mathbf{Z}}\right]=E\left[\exp\left\{ \frac{i\omega}{\sigma\sqrt{n}}\sum_{k=1}^{n}\left(\mathbf{X}_{k}-\mu\right)\right\} \right]=E\left[\prod_{k=1}^{n}e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right] $$ =\prod_{k=1}^{n}E\left[e^{i\omega\left(\mathbf{X}_{k}-\mu\right)/\sigma\sqrt{n}}\right]=\cdots=\left(E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]\right)^{n}. $
We can expand the exponential as a power series (in ω about ω = 0 ).
$ E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right] $
It can be shown that $ \frac{E\left[R\left(\omega\right)\right]}{\omega^{2}/2n}\longrightarrow0 $ as $ n\rightarrow\infty , \forall\omega\in\mathbf{R} $ .
Thus we have
$ E\left[e^{i\omega\left(\mathbf{X}-\mu\right)/\sigma\sqrt{n}}\right]=1-\frac{\omega^{2}}{2n}+E\left[R\left(\omega\right)\right]\backsimeq1-\frac{\omega^{2}}{2n}\text{ as }n\rightarrow\infty. $
$ \Phi_{\mathbf{Z}_{n}}\left(\omega\right)\backsimeq\left(1-\frac{\omega^{2}}{2n}\right)^{n}=\left(1+\frac{\left(-\omega^{2}/2\right)}{n}\right)^{n}\longrightarrow e^{-\omega^{2}/2}\text{ as }n\rightarrow\infty. $
$ \therefore\Phi_{\mathbf{Z}_{n}}\left(\omega\right)\longrightarrow e^{-\omega^{2}/2}\Longrightarrow F_{\mathbf{Z}_{n}}\left(z\right)\longrightarrow\mathbf{\Phi}\left(z\right)=\int_{-\infty}^{z}\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}dx. $
We have “proved” the central limit theorem for i.i.d. random variables. It can be proven under much more general circumstances (e.g. sums of correlated sequences of random variables). Lindeberg-Feller central limit theorem works for some classes of correlated random variables.
Gaussian normalization
Gaussian distribution $ \left(\mathbf{X}\right) $ with μ and σ2⇒ Standard normal distribution $ \left(\mathbf{Z}\right) $ that has mean 0 and variance 1
$ \mathbf{Z}=\frac{\mathbf{X}-\mu}{\sigma}\text{ and }\mathbf{X}=\sigma\mathbf{Z}+\mu. $
