View source for ECE600 F13 rv distribution mhossain - Rhea

[[Category:ECE600]]
[[Category:Lecture notes]]

[[ECE600_F13_notes_mhossain|Back to all ECE 600 notes]]<br/>
[[ECE600_F13_rv_definition_mhossain|Previous Topic: Random Variables: Definitions]]<br/>
[[ECE600_F13_rv_conditional_distribution_mhossain|Next Topic: Conditional Distributions]]
----
[[Category:ECE600]]
[[Category:probability]]
[[Category:lecture notes]]
[[Category:slecture]]

<center><font size= 4>
[[ECE600_F13_notes_mhossain|'''The Comer Lectures on Random Variables and Signals''']]
</font size>

[https://www.projectrhea.org/learning/slectures.php Slectures] by [[user:Mhossain | Maliha Hossain]] 


<font size= 3> Topic 6: Random Variables: Distributions</font size>
</center>

----
----

How do we find, compute and model P(x ∈ A) for a random variable X for all A ∈ B('''R''')? We use three different functions: <br/>
# the cumulative distribution function (cdf)
# the probability density function (pdf)
# the probability mass function (pmf)

We will discuss these in this order, although we could come at this discussion in a different way and a different order and arrive at the same place.

'''Definition''' <math>\quad</math> The '''cumulative distribution function (cdf)''' of X is defined as <br/>
<center><math>\begin{align}
F_X(x) &= P_X((-\infty,x])\;\forall x\in \mathbb R \\
&= P(X^{-1}((-\infty, x])) \\
&=P(\{ \omega\in\mathcal S:\;X(\omega)\leq x \})
\end{align}</math></center>

'''Notation''' <math>\quad</math> Normally, we write this as <br/>
<center><math>F_X(x) = P(X\leq x)</math></center>
So <math>F_X(x)</math> tells us P<math>P_X(A)</math> if A = (-∞,x] for some real x.<br/>
What about other A ∈ B('''R''')?  It can be shown that any A ∈ B('''R''') can be written as a countable sequence of set operations (unions, intersections, complements) on intervals of the form (-∞,x<math>_n</math>], so can use the probability axioms to find <math>P_X(A)</math> from <math>F_X</math> for any A ∈ B('''R'''). This is not how we do things in practice normally. This will be discussed more later. 

Can an arbitrary function <math>F_X,</math> be a valid cdf? No, it cannot. <br/>
Properties of a valid cdf: <br/>
1. <br/>
<center><math> \lim_{x\rightarrow\infty}F_X(x) = 1\;\;\mbox{and}\;\;\lim_{x\rightarrow -\infty}F_X(x) = 0</math></center>
This is because <br/>
<center><math>\lim_{x\rightarrow\infty}F_X(x) = P(\{ \omega\in\mathcal S:\;X(\omega)\leq\infty \})=1</math></center>
and <br/>
<center><math>\lim_{x\rightarrow -\infty}F_X(x)= P(\varnothing)= 0</math></center>

2. For any <math>x_1,x_2</math> ∈ '''R''' such that <math>x_1<x_2</math>, <br/>
<center><math>F_X(x_1)\leq F_X(x_2)</math></center>
i.e. <math>F_X(x)</math> is a non decreasing function.

3. <math>F_X</math> is continuous from the right , i.e. <br/>
<center><math>F_X(x^+) \equiv \lim_{\epsilon\rightarrow0,\epsilon>0}F_X(x+\epsilon)=F_X(x)\;\;\forall x\in\mathbb R</math></center>
'''Proof''':
First, we need some results from analysis and measure theory:<br/>
(i) For a sequence of sets, <math>A_1, A_2,...</math>, if <math>A_1</math> ⊃ <math>A_2</math> ⊃ ..., then <br/>
<center><math>\lim_{n\rightarrow\infty}A_n = \bigcap_{n=1}^{\infty}A_n</math></center>

(ii) If <math>A_1</math> ⊃ <math>A_2</math> ⊃ ..., then <br/>
<center><math>P(\lim_{n\rightarrow\infty}A_n) = \lim_{n\rightarrow\infty}P(A_n)</math></center>

(iii) We can write <math>F_X(x^+)</math> as <br/>
<center><math>F_X(x^+) = \lim_{n\rightarrow\infty}F_X(x+\frac{1}{n})</math></center>
Now let <br/>
<center><math> A_n = \{X\leq x+\frac{1}{n}\}</math></center>
Then <br/>
<center><math>\begin{align}
F_X(x^+) &= \lim_{n\rightarrow\infty}P(X\leq x+\frac{1}{n}) \\
&=\lim_{n\rightarrow\infty}P(A_n) \\
&=P(\lim_{n\rightarrow\infty}A_n) \\
&=P(\bigcap_{n=1}^{\infty}A_n) \\
&=P(\bigcap_{n=1}^{\infty}\{X\leq x+\frac{1}{n}\})\\
&=F_X(x)
\end{align}</math></center>

4. <math>P(X>x) = 1-F_X(x)</math> for all x ∈ '''R'''

5. If <math>x_1 < x_2</math>, then <br/>
<center><math> P(x_1<X\leq x_2) = F_X(x_2) - F_X(x_1)\;\forall x_1,x_2\in\mathbb R</math></center>

6. <math>P(\{X=x\})= F_X(x) - F_X(x^-)</math>, where <br/>
<center><math> F_X(x^-) = \lim_{\epsilon\rightarrow 0,\epsilon>0} F_X(x-\epsilon)</math></center>


----

== The Probability Density Function  ==

'''Definition''' <math>\quad</math> The '''probability density function (pdf)''' of a random variable X is the derivative of the cdf of X, <br/>
<center><math> f_X(x) = \frac{dF_X(x)}{dx}</math></center>
at points where <math>F_x</math> is differentiable. <br/>
From the Fundamental Theorem of Calculus, we then have that <br/>
<center><math>F_X(x)=\int_{-\infty}^xf_X(r)dr\;\;\forall x\in\mathbb R</math></center>

'''Important note:''' the cdf <math>F_X</math> might not be differentiable everywhere. At points where <math>F_X</math> is not differentiable, we can use the Dirac delta function to defing <math>f_x</math>.

'''Definition''' <math>\quad</math> The '''Dirac Delta Function <math>\delta(x)</math>''' is the function satisfying the properties: <br/>
1. <br/> 
<center><math>\delta(x) = 0 \;\forall x\neq0</math></center> 
2. <br/>
<center><math>\int_{-\infty}^{\infty} \delta(x)dx = \int_{-\epsilon}^{\epsilon}\delta(x)dx = 1\;\forall\epsilon>0</math></center>

If <math>F_X</math> is not differentiable at a point, use <math>\delta(x)</math> at that point to represent <math>f_X</math>.

Why do we do this? Consider the step function <math>u(x)</math>, which is discontinuous and thus not differentiable at <math>x=0</math>. This is a common type of discontinuity we see in cdfs. The derivative of <math>u(x)</math> is defined as <br/>
<center><math>\frac{du(x)}{dx}=\lim_{h\rightarrow 0}\frac{u(x+h)-u(x)}{h}</math></center>
This limit does not exist at <math>x=0</math> 

Let's look at the function <br/>
<center><math>g(x) = \frac{u(x+h)-u(h)}{h}</math></center>
It looks like this:

<center>[[Image:fig1_rv_distributions.png|400px|thumb|left|Fig 1: g(x) for h>0]]</center>


For any x ≠ 0, we have that <br/>
<center><math>\frac{u(x+h)-u(x)}{h}=0</math></center>
for small enough h.<br/>
Also, ∀ <math>\epsilon</math> > 0, <br/>
<center><math> \int_{-\epsilon}^{\epsilon}\delta(x)dx= 1\; \forall h<\epsilon</math></center>
So, in the limit, the function g(x) has the properties of the <math>\delta</math>-function as h tends to 0. A similar argument can be made for h<0.<br/>
So this is why it is sometimes written that <br/>
<center><math> \frac{du(x)}{dx} = \delta(x)</math></center>

Since we will only work with non-differentiable functions that have step discontinuities as cdfs, we write <br/>
<center><math>f_X(x) = \frac{dF_X(x)}{dx}</math></center>
with the understanding that <math>d/dx</math> is not necessarily the traditional definition of the derivative.

'''Properties of the pdf:'''<br/>
1. (proof: derivative of increasing Function F<math>_X</math>(x) must be non-negative)<br/>
<center><math> f_X(x)\geq 0\;\forall x\in \mathbb R</math></center> <br/>

2. (proof: use the fact that the limit of F<math>_X</math>(x) as x goes to infinity is 1)<br/>
<center><math>\int_{-\infty}^{\infty}f_X(x)dx = 1</math></center><br/>

3. If <math>x_1<x_2</math>, then<br/>
<center><math>P(x_1<X\leq x_2) = \int_{x_1}^{x_2}f_X(x)dx</math></center>
:(proof: use the fact that P(x<math>_1</math> < X ≤ x<math>_2</math>) = F<math>_X</math>(x<math>_2</math>) - F<math>_X</math>(x<math>_1</math>))<br/>

Some notes:<br/>
* We introduced the concept of a pdf in our discussion of probability spaces. We could have defined the pdf of a random variable X as a function <math>f_X</math> satisfying properties 1 and 2 above, and then define <math>F_X</math> in terms of <math>f_X</math>.
* f_X(x) is not a probability for a fixed x, it gives us instead the "probability density", so it must be integrated to give us the probability. 
* In practice, to compute probabilities of random variable X, we normally use <br/>
<center><math> P(X\in A) = \int_{A}f_X(x)dx</math></center>


----

== Continuous and Discrete Random Variables ==

A random variable X having a cdf that is continuous everywhere is called a '''continuous random variable'''.

<center>[[Image:fig2_rv_distributions.png|400px|thumb|left|Fig:2 a possible cdf for a continuous random variable]]</center>


A random variable X having a piece-wise constant cdf is a '''discrete random variable'''.

<center>[[Image:fig3_rv_distributions.png|400px|thumb|left|Fig:3 a possible cdf for a discrete random variable]]</center>


A random variable X whose cdf is neither continuous everywhere nor piece-wise constant is a '''mixed random variable'''.

<center>[[Image:fig4_rv_distributions.png|400px|thumb|left|Fig:4 a possible cdf for a mixed random variable]]</center>


We will consider only discrete and continuous random variables in this course. 

'''Note:'''
* For a continuous random variable X, P(X=x)=0 ∀x ∈ '''R''', because <math>F_X(x)-F_X(x^-) = 0</math> ∀x ∈ '''R'''.
* We can use the cdf/pdf functions for a discrete random variable X, but generally, we do not. Instead, we use the pmf. 


----

==Probability Mass Function ==

The '''probability mass function (pmf)''' of random variable X is the function <br/>
<center><math>p_X(x) \equiv p_X(\{x\})=P(X=x)</math></center>
We will use this function when X is discrete. 

Although P(X=x) exists for every x ∈ '''R''', we normally define <math>p_X(x)</math> for some subset ''R''⊂'''R'''

'''Definition''' <math>\quad</math> Given random variable X:''S''→'''R''', let ''R'' = X(''S'') be the ''range space'' of X (note that the range of X is still '''R''').

'''Example:''' Let X be the sum of values rolled on two fair die. Then ''R'' = {2,3,...,12}.

We define the pmf only on ''R'', so in the example above, we would have <math>p_X(x) = P(X=x)</math> ∀x ∈ ''R''.

We can now consider the probability space for a discrete random variable to be (''R'',''P(R)'',''p<math>_X</math>). <br/>
Note that if X is continuous, we define the cdf/pdf for all x ∈ '''R'''. for example, if X = V<math>^2</math>, where V is the voltage (so X is proportional to power), then ''R''=[0,∞), but we still define <math>f_X(x)</math> ∀x ∈ ''R''.

Properties of the pmf: <br/>
1. (proof)<br/>
<center><math>p_X(x) \geq 0</math></center>
2. (proof)<br/>
<center><math>\sum_{x\in\mathcal R}p_X(x) = 1</math></center>

These properties can be derived from the axioms. Note that we could simply define the pmf to be a function having these two and then create a probability mapping P(.) in such a way that P(.) will satisfy the axioms. We discussed this in our lectures on probability spaces. This is what we do in practice. 


----

==Some Important Random Variables == 

==1. Gaussian Random Variable ==

<center>[[Image:fig5_rv_distributions.png|400px|thumb|left|Fig:5 The pdf of a random variable that is Gaussian distributed with mean <math>\mu</math> and variance <math>\sigma^2</math>.]]</center>


The Gaussian (or Normal) continuous random variable <math>X</math> has pdf <br/>
<center><math>f_X(x) = \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\;\;\forall x\in\mathbb R;\;\sigma,\mu\in\mathbb R, \sigma>0</math></center>
and cdf <math>F_X(x)</math>, where <br/>
<center><math>F_X(x)=\int_{-\infty}^x\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}dr</math></center>

(No closed solution)<br/>
Let <br/>
<center><math>\phi(x) = \int_{-\infty}^x\frac{1}{\sqrt{2\pi}}e^{-\frac{x^2}{2}}dr</math></center>

Then write <br/>
<center><math>F_X(x) = \phi(\frac{x-\mu}{\sigma})</math></center>

Use the table to find values of ϕ.

Notation for Gaussian random variable <math>X</math>:<math>N(\mu,\sigma^2)</math>

Gaussian random variable are used to model 
* certain types of noise 
* sum of large number of independent random variables
* continuous random variables with no prior information about distribution (default assumption)


==2. Uniform Random Variable ==

'''Continuous Case:'''<br/>
<center><math>f_X(x) = 
\begin{cases}
\frac{1}{x_2-x_1} & \mbox{if }x_1\leq x\leq x_2 \\
0 & \mbox{else}
\end{cases}\quad x_1,x_2 \in\mathbb R;\;\forall x\in\mathbb R
</math></center>


<center>[[Image:fig6_rv_distributions.png|400px|thumb|left|Fig:6 The pdf of a continuous random variable that is uniformly distributed with parameters a=<math>x_1<x_2</math>=b.]]</center>


<center><math>F_X(x) = 
\begin{cases}
1 & \mbox{if }x\geq x_2 \\
\frac{x-x_1}{x_2-x_1} & \mbox{if }x_1\leq x\leq x_2 \\
0 & \mbox{if }x<x_1
\end{cases}
</math></center>

'''Discrete Case'''

<math>R = \{x_1,...,x_2\}</math> for some integer n ≥ 1.<br/>
<center><math>p_X(x_i) = \frac{1}{n}\;\forall x_i\in\mathcal R</math></center>

Used to model:
*Continuous case - random variables where <math>P(X</math>∈''(a,b))'' depends only on <math>b-a</math>, ''∀a,b ∈'' '''R''', <math>x_1</math>≤''a''<''b''≤<math>x_2</math>
* Discrete case - random variables whose values are equally likely to occur.


==3. Binomial Random Variable (Disrete)==

<math>R = \{0,1,...,n\}</math><br/>
<center><math>p_X(k) = {n\choose k}p^k(1-p)^{n-k}\;\;k = 0,...,n</math></center>
p ∈ [0,1], n ≥ 1, n finite.

Used to model number of successes in Bernoulli trials


==4. Exponential (Continuous)==

<center>[[Image:fig7_rv_distributions.png|400px|thumb|left|Fig:7 The pdf of a continuous random variable that is exponentially distributed with parameters <math>x_1<x_2</math>.]]</center>


<center><math>f_X(x) = 
\begin{cases}
\lambda e^{-\lambda x} & \mbox{if }x\geq 0 \\
0 & \mbox{else}
\end{cases}\quad \lambda\in\mathbb R;\;\lambda>0
</math></center>

Used to model 
*times between arrival of customers (or other things)
* lifetimes of devices/systems


==5. Rayleigh (Continuous)==

<center>[[Image:fig8_rv_distributions.png|400px|thumb|left|Fig:8 The pdf of a continuous random variable with a Rayleigh distribution.]]</center>


<center><math>f_X(x) = 
\begin{cases}
\frac{x}{\sigma^2}e^{-\frac{x^2}{2\sigma^2}} & x\geq 0 \\
0 & x<0
\end{cases}
</math></center>

Used to model square root of a sum of squares (e.g. magnitude of complex exponential).


==6. Laplace (Continuous)==

<center><math>f_X(x) = \frac{\alpha}{2}e^{-\alpha|x|};\quad\alpha>0</math></center>

<center>[[Image:fig9_rv_distributions.png|400px|thumb|left|Fig:9 The pdf of a continuous random variable with a Laplace distribution.]]</center>


Used to model prediction errors.


==7. Poisson (Discrete)==

<center><math>\mathcal R = \{0,1,2,...\}</math></center>
<center><math>p_X(k)=e^{-\lambda}\frac{\lambda^k}{k!};\quad\lambda>0,\;k=0,1,2,...</math></center>

Used to model the number of occurrences of events in time or space. 


==8. Geometric (Discrete)==

Form 1:<br/>
<center><math>\mathcal R = \{0,1,2,...\}</math></center><br/>
<center><math>p_X(k)=p(1-p)^k \ </math></center><br/>
<center>or</center> <br/>
Form 2: <br/>
<center><math>\mathcal R = \{1,2,3,...\}</math></center><br/>
<center><math>p_X(k)=p(1-p)^{k-1} \ </math></center><br/>

Used to model number of Bernoulli trials until the occurrence of first success. 


----

== References ==

* [https://engineering.purdue.edu/~comerm/ M. Comer]. ECE 600. Class Lecture. [https://engineering.purdue.edu/~comerm/600 Random Variables and Signals]. Faculty of Electrical Engineering, Purdue University. Fall 2013.


----

==[[Talk:ECE600_F13_rv_distribution_mhossain|Questions and comments]]==

If you have any questions, comments, etc. please post them on [[Talk:ECE600_F13_rv_distribution_mhossain|this page]]


----


[[ECE600_F13_notes_mhossain|Back to all ECE 600 notes]]<br/>
[[ECE600_F13_rv_definition_mhossain|Previous Topic: Random Variables: Definitions]]<br/>
[[ECE600_F13_rv_conditional_distribution_mhossain|Next Topic: Conditional Distributions]]