Contents
1.4 Discrete Random Variables
From the ECE600 Pre-requisites notes of Sangchun Han, ECE PhD student.
1.4.1 Bernoulli distribution
$ \mathbf{X}=\begin{cases} \begin{array}{lll} 1 & & ,\text{ if success}\\ 0 & & ,\text{ if fail.} \end{array}\end{cases} $
$ P\left(\left\{ \mathbf{X}=1\right\} \right)=p $.
$ P\left(\left\{ \mathbf{X}=0\right\} \right)=q\left(=1-p\right) $.
$ E\left[\mathbf{X}\right]=1\cdot P\left(\left\{ \mathbf{X}=1\right\} \right)+0\cdot P\left(\left\{ \mathbf{X}=0\right\} \right)=1\cdot p+0\cdot q=p $.
$ E\left[\mathbf{X}^{2}\right]=1^{2}\cdot P\left(\left\{ \mathbf{X}=1\right\} \right)+0^{2}\cdot P\left(\left\{ \mathbf{X}=0\right\} \right)=p $.
$ Var\left[\mathbf{X}\right]=E\left[\mathbf{X}^{2}\right]-\left(E\left[\mathbf{X}\right]\right)^{2}=p-p^{2}=p\left(1-p\right)=pq $.
Moment generating function
$ \phi_{\mathbf{X}}\left(s\right)=E\left[e^{s\mathbf{X}}\right]=e^{s\cdot1}\cdot p+e^{s\cdot0}\cdot q=p\cdot e^{s}+q $.
1.4.2 Binomial distribution
If $ \mathbf{Y}_{1},\mathbf{Y}_{2},\cdots $ are i.i.d. Bernoulli random variables, then Binomial random variable is defined as $ \mathbf{X}=\mathbf{Y}_{1}+\mathbf{Y}_{2}+\cdots+\mathbf{Y}_{n} $, which represents the number of success from $ n $ Bernoulli trials.
$ p_{\mathbf{X}}\left(k\right)=\left(\begin{array}{c} n\\ k \end{array}\right)p^{k}\left(1-p\right)^{n-k} $.
$ E\left[\mathbf{X}\right]=np $.
$ Var\left[\mathbf{X}\right]=np\left(1-p\right)=npq $.
Moment generating function
The moment generating function for Binomial distribution must be $ \phi_{\mathbf{X}}\left(s\right)=\left(p\cdot e^{s}+q\right)^{n} $ because Binomial distribution is the $ n $ convolution of Bernoulli distribution.
Check
$ \phi_{\mathbf{X}}\left(s\right)=E\left[e^{s\mathbf{X}}\right]=\sum_{k=0}^{n}e^{s\cdot k}\left(\begin{array}{c} n\\ k \end{array}\right)p^{k}\left(1-p\right)^{n-k}=\left(p\cdot e^{s}+\left(1-p\right)\right)^{n}=\left(p\cdot e^{s}+q\right)^{n}. $
1.4.3 Geometric distribution
It is defined as the number of Bernoulli trials until success. Geometric random variable is memoryless.
$ p_{\mathbf{X}}\left(k\right)=q^{k-1}p $.
Moment generating function
$ \phi_{\mathbf{X}}\left(s\right)=E\left[e^{s\mathbf{X}}\right]=\sum_{k=1}^{\infty}e^{s\cdot k}q^{k-1}p=\frac{p}{q}\sum_{k=1}^{\infty}\left(e^{s}\cdot q\right)^{k}=\frac{p}{q}\cdot\frac{q\cdot e^{s}}{\left(1-q\cdot e^{s}\right)}=\frac{p\cdot e^{s}}{1-q\cdot e^{s}} $.
Probability generating function
$ P_{\mathbf{X}}\left(z\right)=E\left[z^{\mathbf{X}}\right]=\sum_{k=1}^{\infty}z^{k}\cdot q^{k-1}p=\frac{p}{q}\sum_{k=1}^{\infty}\left(z\cdot q\right)^{k}=\frac{p}{q}\cdot\frac{z\cdot q}{1-z\cdot q}=\frac{z\cdot p}{1-\cdot z\cdot q} $.
Approach 1 for $ E\left[\mathbf{X}\right] $ and $ Var\left[\mathbf{X}\right] $
$ E\left[\mathbf{X}\right]=\sum_{k=1}^{\infty}k\cdot q^{k-1}p=\frac{p}{q}\sum_{k=1}k\cdot q^{k}=\frac{p}{q}\cdot\frac{q}{\left(1-q\right)^{2}}=\frac{p}{p^{2}}=\frac{1}{p} $.
$ E\left[\mathbf{X}^{2}\right]=\sum_{k=1}^{\infty}k^{2}\cdot q^{k-1}p=\frac{p}{q}\sum_{k=1}^{\infty}k^{2}\cdot q^{k}=\text{we cannot solve this equation!} $
Approach 2 for $ E\left[\mathbf{X}\right] $ and $ Var\left[\mathbf{X}\right] $
$ E\left[\mathbf{X}\right] $
$ E\left[\mathbf{X}^{2}\right] $
$ Var\left[\mathbf{X}\right]=E\left[\mathbf{X}^{2}\right]-\left(E\left[\mathbf{X}\right]\right)^{2}=\frac{2-p}{p^{2}}-\frac{1}{p^{2}}=\frac{1-p}{p^{2}} $.
Approach 3 for $ E\left[\mathbf{X}\right] $ and $ Var\left[\mathbf{X}\right] $
$ E\left[\mathbf{X}\right] $
$ E\left[\mathbf{X}^{2}\right] $
$ Var\left[\mathbf{X}\right]=E\left[\mathbf{X}^{2}\right]-\left(E\left[\mathbf{X}\right]\right)^{2}=\frac{2-p}{p^{2}}-\frac{1}{p^{2}}=\frac{1-p}{p^{2}} $.
1.4.4 Poisson distribution when mean and variance are $ \lambda $
$ p(k)=\frac{e^{-\lambda}\lambda^{k}}{k!} $.
$ \Phi_{\mathbf{X}}(\omega)=\sum_{k=0}^{\infty}\frac{e^{-\lambda}\lambda^{k}}{k!}\cdot e^{i\omega k}=e^{-\lambda}\sum_{k=0}^{\infty}\frac{\left(\lambda\cdot e^{i\omega}\right)^{k}}{k!}=e^{-\lambda}\cdot e^{\lambda e^{i\omega}}=e^{-\lambda\left(1-e^{i\omega}\right)} $.
$ E\left[\mathbf{X}\right]=Var\left[\mathbf{X}\right]=\lambda $.
$ E\left[\mathbf{X}^{2}\right]=Var\left[\mathbf{X}\right]+\left(E\left[\mathbf{X}\right]\right)^{2}=\lambda+\lambda^{2} $.
