Back to all ECE 600 notes

The Comer Lectures on Random Variables and Signals

Slectures by Maliha Hossain


Topic 14: Joint Expectation



Joint Expectation

Given random variables X and Y, let Z = g(X,Y) for some g:R$ _2 $→R. Then E[Z] can be computed using f$ _Z $(z) or p$ _Z $(z) in the original definition of E[ ]. Or, we can use

$ E[Z] = \int\int_{\mathbb R^2}g(x,y)f_{XY}(x,y)dxdy $

or

$ E[Z]=\sum_{x\in\mathcal R_x}\sum_{y\in\mathcal R_y}g(x,y)p_{XY}(x,y) $

The proof is in Papoulis.

We will use joint expectation to define some important moments that help characterize the joint behavior of X and Y.

Note that joint expectation is a linear operator, so if g$ _1 $,...,g$ _n $ are n functions from R$ ^2 $ to R and a$ _1 $,...,a$ _n $ are constants, then

$ E\left[\sum_{i=1}^na_ig_i(X,Y)\right]=\sum_{i=1}^na_iE[g_i(X,Y)] $



Important Moments for X,Y

We still need $ \mu_X $, $ \mu_Y $, $ \sigma_X $$ ^2 $, $ \sigma_X $$ ^2 $, the means and variances of X and Y. Other moments that are of great interest are:

  • The correlation between X and Y is
$ \mbox{Corr}(X,Y)\equiv E[XY] $
  • The covariance of X and Y
$ \mbox{Cov}(X,Y)\equiv E[(X-\mu_X)(Y-\mu_Y)] $
  • The correlation coefficient of X and Y is
$ r_{XY}\equiv\frac{\mbox{Cov}(X,Y)}{\sigma_X\sigma_Y} $
  • Note:
    • |r$ _{XY} $| ≤ 1 (proof)
    • If X and Y are independent, then r$ _{XY} $ = 0. The converse is not true in general.
  • If r$ _{XY} $=0, then X and Y are said to be uncorrelated. It can be shown that
    • X and Yare uncorrelated iff Cov(X,Y)=0 (proof).
    • X and Y are uncorrelated iff E[XY] = $ \mu_X\mu_Y $ (proof).
  • X and Y are orthogonal if E[XY]=0.



The Cauchy Schwarz Inequality

For random variables X and Y,

$ |E[XY]|\leq\sqrt{E[X^2]E[Y^2]} $

with equality iff Y = a$ _0 $X with probability 1, where a$ _0 $ is a constant. Note that "equality with probability 1" will be defined later.

Proof: $ \qquad $ We start by considering

$ 0\leq E[(aX-Y)^2]=a^2E[X^2]-2aE[XY]+E[Y^2] $

which is a quadratic function of a ∈ R. Consider two cases:

(i)$ \; $ E[(aX-Y)^2] > 0
(ii) E[(aX-Y)^2] = 0

Case (i):

$ E[X^2]a^2-2E[XY]a+E[Y^2]>0 \ $


Fig 1: A possible depiction of the quadratic when the discriminant is greater than zero.

Since this quadratic is greater than 0 for all a, there are no real roots. There must be two complex roots. From the quadratic equation, this means that

$ E[XY] < \sqrt{E[X^2]E[Y^2]} $

Case (ii):

$ E[(aX-Y)^2]=0 \ $

In this case

$ E[X^2]a^2-2E[XY]a+E[Y^2]=0 \ $


Fig 2: A possible depiction of the quadratic when the discriminant is equal to zero.


This means that ∃a ∈ R such that

$ E[(a_0X-Y)^2] = 0 \ $

It can be shown that if a random variable X has E[X$ ^2 $]=0, then X=0 except possibly on a set of probability 0. Note that previously, we have defined equality between random variables X and Y to mean

$ X=Y\;\mbox{iff}\;X(\omega)=Y(\omega)\;\forall\omega\in\mathcal S $

With the same notion of equality, we would have that

$ X=0\;\mbox{iff}\;X(\omega)=0\;\forall\omega\in\mathcal S $

However, the result for the case E[X$ ^2 $]=0 is that X=0 with probability 1, which means that there is a set A = {$ \omega $S: X($ \omega $) = 0} with P(A) = 1.

Returning to the case E[(a$ _0 $X-Y)$ ^2 $]=0, we get that Y = a$ _0 $X for some a$ _0 $R, with probability 1.
Thus we have shown that

$ E[XY]=\sqrt{E[X^2]E[Y^2]} $

with equality iff Y = a$ _0 $X with probability 1.



The Correlation Coefficient for Jointly Gaussian Random Variables X and Y

We have previously seen that the joint density function for Gaussian X and Y contains a parameter r. It can be shown that this r is the correlation coeffecient of X and Y:

$ r = \frac{E[XY]-\mu_X\mu_Y}{\sigma_X\sigma_Y} $

Note that one way to show this is to use a concept called iterated expectation, which we will cover later.



Joint Moments

Definition $ \qquad $ The joint moments of X and Y are

$ \mu_{jk}\equiv E[X^jY^k] $

and the joint central moments are

$ \sigma_{jk}=E[(X-\overline{X})^j(Y-\overline{Y})^k] $

for j = 0,1,...; k = 0,1,...



Joint Characteristic Function

Definition $ \qquad $ The joint characteristic function of X and Y is

$ \Phi_{XY}(\omega_1,\omega_2)\equiv E[e^{i(\omega_1X+\omega_2Y)}] $

for $ \omega_1,\omega_2 $R.

If X and Y are continuous, we write this as

$ \Phi_{XY}(\omega_1,\omega_2)=\int\int_{\mathbf R^2}e^{i(\omega_1x+\omega_2y)}f_{XY}(x,y)dxdy $

If X and Y are discrete, we use

$ \Phi_{XY}(\omega_1,\omega_2)=\sum_{x\in\mathcal R_x}\sum_{y\in\mathcal R_y}e^{i(\omega_1x+\omega_2y)}p_{XY}(x,y) $

Note:

$ \bullet\; \Phi_X(\omega)=\Phi_{XY}(\omega,0);\quad\Phi_Y(\omega)=\Phi_{XY}(0,\omega) $
(proof)
$ \bullet\; Z = aX+bY \;\Rightarrow\;\Phi_Z(\omega) =\Phi_{XY}(a\omega,b\omega) $
(proof)
$ \bullet\; X \perp\!\!\!\perp Y \;\mbox{iff}\;\Phi_{XY}(\omega_1,\omega_2)=\Phi_X(\omega_1)\Phi_Y(\omega_2) $
(proof)
$ \bullet\; X \perp\!\!\!\perp Y \;\mbox{and}\;Z=X+Y\;\Rightarrow\Phi_{Z}(\omega)=\Phi_X(\omega)\Phi_Y(\omega) $
(proof)



Moment Generating Function

The joint moment generating function of X and Y is

$ \phi_{XY}(s_1,s_2)=E[e^{s_1X+s_2Y}];\qquad s_1,s_2\in\mathbb C $

Then it can be shown that

$ \begin{align} \mu_{jk}&=E[X^jY^k] \\ &=\frac{\partial^j\partial^k}{\partial s_1^j\partial s_2^k}\phi_{XY}(s_1,s_2)|_{s_1=0,s_2=0} \end{align} $



References



Questions and comments

If you have any questions, comments, etc. please post them on this page



Back to all ECE 600 notes

Alumni Liaison

Correspondence Chess Grandmaster and Purdue Alumni

Prof. Dan Fleetwood