Difference between revisions of "ECE600 F13 Statistical Independence mhossain" - Rhea

Latest revision as of 11:11, 21 May 2014

Back to all ECE 600 notes
Previous Topic: Conditional Probability
Next Topic: Random Variables: Definition

Definition

People generally have an idea of the concept of independence, but we will formalize it for probability theory here.

Definition $ \qquad $ Given (S,F,P) and A, B ∈ F, events A and B are statistically indendent iff

It may seem more intuitive if we consider that, from the definition above, we can say that A and B are statistically independent iff P(A|B)P(B) = P(A)P(B), which means P(A|B) = P(A)

Notation $ \qquad $ A, B are independent ⇔

Note $ \qquad $ We will usually refer to statistical independence as simply independence in this course.

Note $ \qquad $ If A and B are independent, then A' and B are also independent.

Proof:

Definition $ \qquad $ Events $ A_1,A_2,...,A_n $ are statistically independent iff for any $ k=1,2,...,n $, and any 1 ≤ $ j_1,j_2,...,j_k $ ≤ n, for any finite n, we have that

Note that to show that a collection of sets is disjoint, we only need to consider pairwise intersections of the sets. But to show that a collection of sets is independent, we need to consider every possible combination of sets from the collection. Then we say that the events are mutually independent. Mutual independence implies pairwise independence but the converse is not true.
Note that for a given n, there are $ 2^n-(n+1) $ combinations. To see this, use the Binomial Theorem:

Then

So,

So there are $ 2^n-(n+1) $ combinations of sets that need to be checked for independence.
In practice, generally we do not check $ 2^n-(n+1) $ combinations for every n. Often, we just assume independence based on intuition and see if results are reasonable. We cannot however, use this approach in this class.

Combined Experiments

Consider two experiments with probability spaces ($ S_1,F_1, $P$ _1 $) and ($ S_2,F_2, $P$ _2 $). Consider a combined experiment with probability space (S,F,P), where

where S is the cartesian product of $ S_1 $ and $ S_2 $.
What should F be?
Let

Or, let

Note that for discrete S, these are typically the same. But ∃ subsets of S = $ S_1 $×$ S_2 $ that are not in F for both of these definitions, some of which might be events we care about.

Example
Experiment 1: Toss a coin
Experiment 2: Roll a die

If we let

then, the event $ \{(H,2),(T,6)\} $ is not in F, since $ \{(H,2),(T,6)\} $ ≠ $ AxB $ for any A ∈ $ F_1 $, B ∈ $ F_2 $.

Instead, we let

This F contains the events we want.

For example, consider $ \{(H,2),(T,6)\} $. Let A = {H} ∈ $ F_1 $, B = {2} ∈ $ F_2 $, C = {T} ∈ $ F_1 $, D = {6} ∈ $ F_2 $
Then, $ AxB $={(H,2)} ∈ F and $ CxD $={(T,6)} ∈ F.
Since F is a $ \sigma $-field, {(H,2)} ∪ {(T,6)} = {(H,2),(T,6)} ∈ F.

Now, what about P? We need P(C) ∀C ∈ F. Start with $ P(AxS_2)=P_1(A) $ ∀A ∈ $ F_1 $ since $ P(AxS_2) $ is the event that A occurs in Experiment 1 and a "don't care" in Experiment 2.
Also, let $ P(S_1xB)=P_2(B) $ ∀B ∈ $ F_2 $.
What about other Cs?
We will assume that the two experiments are independent in the sense that $ A $∈$ F_1 $, $ B $∈$ F_2 $

Then, we have

So now we have that P(C) if

• C = Ax$ S_2 $ or C = $ S_1 $xB for some A ∈ $ F_1 $ or B ∈ $ F_2 $
• C = AxB for A ∈ $ F_1 $, B ∈ $ F_2 $.

What about other Cs?
Any $ C $ that cannot be written as $ AxB $ for some A ∈ $ F_1 $ and B ∈ $ F_2 $ can be written in terms of unions, intersections and complements of such $ AxB $ sets, so the probability axioms can be used to define $ P(C) $ in that case.
Note that the above discussion can be easily extended for n combined experiments.

Bernoulli Trials

Bernoulli trials are an important combined experiment that models repeated independent trials of an experiment with one particular event of interest. Consider an experiment with probability space ($ S_0,F_0, $P$ _0 $), and an event A ∈ $ F_0 $ with P(A) = p. For this combined experiment, let S = $ S_0 $ × ... × $ S_0 $ (n terms) ⇒ S = {n-tuples with each component an element of $ S_0 $}, where n is the number of trials. We are interested in the probability that A occurs k times in the n trials for k = 0,1,...,n.
Let $ p_n(k) $ ≡ P(A occurs k times in n trials).
Then, it can be shown that

Overview of proof: From combinatorics, There are $ ^nC_k $ (n choose k) ways to have k successes (A occurs) in n trials. Due to the independence of the trials, the probability that a give n combination occurs is $ p^k(1-p)^{n-k} $. Since the collection of all combinations is disjoint,

Example:
Consider a binary communication system.
Let A = {bit error} = {send 0, decode 1} ∪ {send 1, decode 0}
Let p = P(A) = $ 10^{-3} $
Transmit each bit 3 times and use the majority vote to decode.
Then, P(wrong decision) = P(2 or more bit errors) = $ p_3(2)+p_3(3) $, which is approximately $ 3 x 10^{-6} $

References

• M. Comer. ECE 600. Class Lecture. Random Variables and Signals. Faculty of Electrical Engineering, Purdue University. Fall 2013.

Questions and comments

If you have any questions, comments, etc. please post them on this page

Back to all ECE 600 notes
Previous Topic: Conditional Probability
Next Topic: Random Variables: Definition