ECE600 F13 Expectation mhossain - Rhea

Topic 9: Expectation

Thus far, we have learned how to represent the probabilistic behavior of a random variable X using the density function f $$ _X $$ or the mass function p $$ _X $$ .
Sometimes, we want to describe X probabilistically using only a small number of parameters. The expectation is often used to do this.

Definition $\qquad$ the expected value of continuous random variable X is defined as

E[X] = \int_{-\infty}^{\infty}xf_X(x)dx

Definition $\qquad$ the expected value of discrete random variable X is defined as

E[X] = \sum_{x\in\mathcal R_X}xp_X(x)

where $$ R_X $$ is the range space of X.

Note:

E[X] is also known as the mean of X. Other notation for E[X] include:

EX,\;\overline{X},\;m_X,\;\mu_X

The equation defining E[X] for discrete X could have been derived from that for continuous X, using the density function f $$ _X $$ containing $\delta$ -functions.

Example $\qquad$ X is an exponential random variable. find E[X].

f_X(x) = \lambda e^{-\lambda x}u(x) \

\begin{align} E[X] &= \int_{-\infty}^{\infty}xf_X(x)dx \\ &= \int_{0}^{\infty}x\lambda e^{-\lambda x}dx \\ &= \frac{1}{\lambda} \end{align}

Let $\mu = 1/\lambda$ . We often write

f_X(x) = \frac{1}{\mu} e^{-\frac{1}{\mu}x}u(x) \

Example $\qquad$ X is a uniform discrete random varibable with $$ R_X $$ = {1,...,n}. Then,

\begin{align} E[X]&=\sum_{k=1}^n\frac{k}{n} \\ &=\frac{1}{n}\sum_{k=1}^n k \\ \\ &= \frac{1}{n}\left(\frac{1}{2}\right)(n)(n+1) \\ \\ &=\frac{n+1}{2} \end{align}

Having defined E[X], we will now consider more general E[g(X)] for a function g:R → R.

Let Y = g(X). What is E[Y]? From previous definitions:

E[Y]=\int_{-\infty}^{\infty}yf_Y(y)dy

or

E[Y] = \sum_{y\in\mathcal R_Y}yp_Y(y)

We can find the expectation of Y by first finding f $$ _Y $$ or p $$ _Y $$ in terms of g and f $$ _X $$ or p $$ _X $$ . Alternatively, it can be shown that

E[Y]=E[g(X)]=\int_{-\infty}^{\infty}g(x)f_X(x)dx

or

E[Y] = E[g(X)]=\sum_{y\in\mathcal R_X}g(x)p_X(x)

See Papoulis for a proof of the above.

Two important cases or functions g:

g(x) = x. Then E[g(X)] = E[X]
g(x) = (x - $\mu_X)^2$ . Then E[g(X)] = E[(X - $\mu_X)^2$ ]

E[g(X)] = \int_{-\infty}^{\infty}(x-\mu_X)^2f_X(x)dx

or

E[g(X)] = \sum_{x\in\mathcal R_x}(x-\mu_X)^2p_X(x)

Note: $\qquad$ E[(X - $\mu_X)^2$ ] is called the variance of X and is often denoted $\sigma_X$ $$ ^2 $$ . The positive square root, denoted $\sigma_X$ , is called the standard deviation of X.

Important property of E[]:
Let g $$ _1 $$ :R → R; g $$ _2 $$ :R → R; $\alpha,\beta$ ∈ R. Then

E[\alpha g_1(X) +\beta g_2(X)] = \alpha E[g_1(X)]+\beta E[g_2(X)] \

So E[] is a linear operator. The proof follows from the linearity of integration.

Important property of Var():

Var(X) = E[X^2]-\mu_X^2

Proof:

\begin{align} E[(X-\mu_X)^2]&=E[X^2-2X\mu_X+\mu_X^2] \\ &=E[X^2]-2\mu_XE[X]+E[\mu_X^2] \\ &=E[X^2]-2\mu_X^2+\mu_X^2 \\ &=E[X^2]-\mu_X^2 \end{align}

Example $\qquad$ X is Gaussian N( $\mu,\sigma^2$ ). Find E[X] and Var(X).

E[X] = \int_{-\infty}^{\infty}\frac{x}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}dx

Let r = x - $\mu$ . Then

E[X] = \int_{-\infty}^{\infty}\frac{r}{\sqrt{2\pi}\sigma}e^{-\frac{r^2}{2\sigma^2}}dr\;+\; \mu\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{r^2}{2\sigma^2}}dr

First term: Integrating an odd function over (-∞,∞) ⇒ first term is 0.
Second term: Integrating a Gaussian pdf over (-∞,∞) gives one ⇒ second term is $\mu$ .
So E[X] = $\mu$

E[X^2] = \int_{-\infty}^{\infty}\frac{x^2}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}dx

Using integration by parts (proof), we see that this integral evaluates to $\sigma^2+\mu^2$ . So,

Var(X) = \sigma^2+\mu^2-\mu^2 = \sigma^2

Example $\qquad$ X is Poisson with parameter $\lambda$ . Find E[X] and Var(X).

\begin{align} E[X] &= \sum_{k=0}^{\infty}k\frac{e^{-\lambda}\lambda^k}{k!} \\ &= \sum_{k=1}^{\infty}\frac{e^{-\lambda}\lambda^k}{(k-1)!} \\ &= \lambda\sum_{k=0}^{\infty}e^{-\lambda}\frac{\lambda^k}{k!} \\ &= \lambda \end{align}

\begin{align} E[X^2] &= \sum_{k=0}^{\infty}k^2\frac{e^{-\lambda}\lambda^k}{k!} \\ &= \sum_{k=0}^{\infty}(k+1)\frac{e^{-\lambda}\lambda^{k+1}}{k!} \\ &= \lambda\sum_{k=0}^{\infty}\frac{ke^{-\lambda}\lambda^k}{k!}\;+\;\lambda\sum_{k=0}^{\infty}\frac{e^{-\lambda}\lambda^k}{k!} \\ \\ &= \lambda E[X] + \lambda(1) \\ &= \lambda^2+\lambda \end{align}

So,
$E[X^2] = \lambda^2 +\lambda \$
$\Rightarrow Var(X) = \lambda^2 +\lambda - \lambda^2 = \lambda \$

Moments

Moments generalize mean and variance to nth order expectations.

Definition $\qquad$ the nth moment of random variable X is

\mu_n\equiv E[X^n]=\int_{-\infty}^{\infty}x^nf_X(x)dx\quad n=1,2,...

and the nth central moment of X is

v_n\equiv E[(X-\mu_X)^n] = \int_{-\infty}^{\infty}(x-\mu_X)^nf_X(x)dx\qquad n = 2,3,...

So

$\mu_1$ = E[X] mean
$\mu_2$ = E[X $$ ^2 $$ ] mean-square
v $$ _2 $$ = Var(X) variance

Conditional Expectation

For an event M ∈ F with P(M) > 0.

E[g(X)|M] = \int_{-\infty}^{\infty}g(x)f_X(x|M)dx

or

E[g(X)|M] = \sum_{x\in\mathcal R_x}g(x)p_X(x|M)dx

Example $\qquad$ X is an exponential random variable. Let M = {X > $\mu$ }. Find E[X|M]. Note that P(M) = P(X > $\mu$ ) and since $\mu$ > 0,

P(M) = P(X>\mu) =\int_{\mu}^{\infty}\frac{1}{\mu}e^{-\frac{x}{\mu}}dx \;>\;0

It can be shown that

f_X(x|X>\mu) = \frac{1}{\mu}e^{-\frac{x-\mu}{\mu}}u(x-\mu)

Then,

\begin{align} E[X|X>\mu] &=\int_{\mu}^{\infty}\frac{x}{\mu}e^{-\frac{x-\mu}{\mu}} \\ &=2\mu \end{align}

Fig 1: Conditional Expectation; X is exponential.

References

M. Comer. ECE 600. Class Lecture. Random Variables and Signals. Faculty of Electrical Engineering, Purdue University. Fall 2013.

Questions and comments

If you have any questions, comments, etc. please post them on this page

Back to all ECE 600 notes
Previous Topic: Functions of a Random Variable
Next Topic: Characteristic Functions

ECE600 F13 Expectation mhossain - Rhea

Contents

Moments

Conditional Expectation

References

Questions and comments

Alumni Liaison