1.10 Two Random Variables

Fact

Let $ \mathbf{X} $ and $ \mathbf{Y} $ be two jointly-distributed, statistically independent random variables, having pdfs $ f_{\mathbf{X}}\left(x\right) $ and $ f_{\mathbf{Y}}\left(y\right) $ . Then the pdf of $ \mathbf{Z}=\mathbf{X}+\mathbf{Y} $ is

$ f_{\mathbf{Z}}\left(z\right)=\left(f_{\mathbf{X}}*f_{\mathbf{Y}}\right)\left(z\right)=\int_{-\infty}^{\infty}f_{\mathbf{X}}\left(x\right)f_{\mathbf{Y}}\left(z-x\right)dx=\int_{-\infty}^{\infty}f_{\mathbf{Y}}\left(y\right)f_{\mathbf{X}}\left(z-y\right)dy $.

The discrete version of above equation is

$ P\left(\mathbf{Z}=z\right)=\sum_{k=-\infty}^{\infty}P\left(\mathbf{X}=k\right)P\left(\mathbf{Y}=z-k\right)=\sum_{k=-\infty}^{\infty}P\left(\mathbf{X}=z-k\right)P\left(\mathbf{Y}=k\right) $.

Example. Sum of two exponential random variables

Let $ \mathbf{X} $ and $ \mathbf{Y} $ be two jointly-distributed RVs having exponential distributions with mean $ \mu $ . Let $ \mathbf{Z}=\mathbf{X}+\mathbf{Y} $ . Find $ f_{\mathbf{Z}}\left(z\right) $ .

$ f_{\mathbf{Z}}\left(z\right) $

Joint Gaussian pdf

$ f_{\mathbf{XY}}(x,y)=\frac{1}{2\pi\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}\sqrt{1-r^{2}}}\exp\left\{ \frac{-1}{2\left(1-r^{2}\right)}\left[\frac{\left(x-\mu_{\mathbf{X}}\right)^{2}}{\sigma_{\mathbf{X}}^{2}}-\frac{2r\left(x-\mu_{\mathbf{X}}\right)\left(y-\mu_{\mathbf{Y}}\right)}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}+\frac{\left(y-\mu_{\mathbf{Y}}\right)^{2}}{\sigma_{\mathbf{Y}}^{2}}\right]\right\} $

If $ r=0 $ ($ \mathbf{X} $ and $ \mathbf{Y} $ are uncorrelated),

$ f_{\mathbf{XY}}(x,y) $

$ \Longrightarrow\mathbf{X} $ and $ \mathbf{Y} $ are independent.

If two RVs $ \mathbf{X} $ and $ \mathbf{Y} $ are uncorrelated Gaussian, then $ \mathbf{X} $ and $ \mathbf{Y} $ are independent. This is the exception of “$ \mathbf{X} $ and $ \mathbf{Y} $ are independent $ \left(\nLeftarrow\right)\Rightarrow \mathbf{X} $ and $ \mathbf{Y} $ are uncorrelated”.

Definition. Uncorrelation

Two RVs \mathbf{X} and \mathbf{Y} are uncorrelated if their covariance is equal to zero. This is true if any of the following three equivalent condition is true:

1. $ Cov\left(\mathbf{X},\mathbf{Y}\right)=0 $

2. $ r_{\mathbf{XY}}=0 $

3. $ E\left[\mathbf{XY}\right]=E\left[\mathbf{X}\right]E\left[\mathbf{Y}\right] $

Note

• $ \mathbf{X} $ and $ \mathbf{Y} $ are uncorrelated $ \Longleftrightarrow E\left[\mathbf{XY}\right]=E\left[\mathbf{X}\right]\cdot E\left[\mathbf{Y}\right]\Longleftrightarrow Cov\left(\mathbf{X},\mathbf{Y}\right)=0 $

• $ \mathbf{X} $ and $ \mathbf{Y} $ are independent $ \Longleftrightarrow f_{\mathbf{XY}}(x,y)=f_{\mathbf{X}}(x)\cdot f_{\mathbf{Y}}(y) $

• $ \mathbf{X} $ and $ \mathbf{Y} $ are independent $ \left(\nLeftarrow\right)\Longrightarrow \mathbf{X} $ and $ \mathbf{Y} $ are uncorrelated Definition. Orthogonality

Two RVs are orthogoal if $ E\left[\mathbf{XY}\right]=0 $ .

Fact

If $ E\left[\mathbf{X}^{2}\right]<\infty and E\left[\mathbf{Y}^{2}\right]<\infty , then \left|E\left[\mathbf{XY}\right]\right|\leq\sqrt{E\left[\mathbf{X}\right]^{2}E\left[\mathbf{Y}\right]^{2}} $ with equality iff $ \mathbf{Y}=a_{0}\mathbf{X} $ where $ a_{0} $ is constant.

Recall

For a quadratic equation $ ax^{2}+bx+c=0,\; a\neq0 $ , the discriminant is $ b^{2}-4ac $ .

Proof

$ E\left[\left(a\mathbf{X}-\mathbf{Y}\right)^{2}\right]\geq0\Longrightarrow a^{2}E\left[\mathbf{X}^{2}\right]-2aE\left[\mathbf{XY}\right]+E\left[\mathbf{Y}^{2}\right]\geq0. $

n.b. LHS is a quadratic in a .

Let's consider two cases: $ (i)\; E\left[\left(a\mathbf{X}-\mathbf{Y}\right)^{2}\right]>0 , (ii)\; E\left[\left(a\mathbf{X}-\mathbf{Y}\right)^{2}\right]=0 $ .

(i) $ 0<E\left[\left(a\mathbf{X}-\mathbf{Y}\right)^{2}\right]=a^{2}E\left[\mathbf{X}^{2}\right]-2aE\left[\mathbf{XY}\right]+E\left[\mathbf{Y}^{2}\right] $ $ \Longrightarrow $ quadratic in a has complex roots $ \Longrightarrow $ “discriminant” of this quadratic is negative

$ 4a^{2}E\left[\mathbf{XY}\right]^{2}-4a^{2}E\left[\mathbf{X}^{2}\right]E\left[\mathbf{Y}^{2}\right] $

(ii) $ 0=E\left[\left(a\mathbf{X}-\mathbf{Y}\right)^{2}\right]=a^{2}E\left[\mathbf{X}^{2}\right]-2aE\left[\mathbf{XY}\right]+E\left[\mathbf{Y}^{2}\right] $ for some $ a=a_{0} $ .

n.b. The discriminant is equal to $ 0 $ when $ \mathbf{Y}=a_{0}\mathbf{X} $ .