Suppose I have a fair coin which I toss multiple times.
I want to model a situation where the results of previous flips do not affect my beliefs about the likelihood of heads in the next flip.
And I would like to describe this situation by saying that the coin tosses are independent.
You may say, we already defined the notion of independent events.
Doesn’t this notion apply? Well not quite.
We defined independence of two events.
But here, we want to talk about independence of a collection of events.
For example, we would like to say that the events, heads in the first toss, heads in the second toss, heads in the third toss, and so on, are all independent.
What is the right definition? Let us start with intuition.
We will say that a family of events are independent if knowledge about some of the events doesn’t change my beliefs, my probability model, for the remaining events.
For example, if I want to say that events A1, A2 and so on are independent, I would like relations such as the following to be true.
The probability that event A3 happened and A4 does not happen remains the same even if I condition on some information about some other events.
Let’s say if I tell you that A1 happens or that both A2 happened and A5 did not happen.
The important thing to notice here is that the indices involved in the event of interest are distinct from the indices associated with the events on which I’m given some information.
I’m given some information about the events A1, A2, and A5, what happened to them.
And this information does not affect my beliefs about something that has to do with events A3 and A4.
I would like all relations of this kind to be true.
This could be one possible definition, just saying that the family of events are independent if and only if any relation of this type is true.
But such a definition would not be aesthetically pleasing.
Instead, we introduce the following definition, which mimics or parallels our earlier definition of independence of two events.
We will say that a collection of events are independent if you can calculate probabilities of intersections of these events by multiplying individual probabilities.
And this should be possible for all choices of indices involved and for any number or events involved.
Let us translate this into something concrete.
Consider the case of three events, A1, A2, and A3.
Our definition requires that we can calculate the probability of the intersection of two events by multiplying individual probabilities.
And we would like all of these three relations to be true, because this property should be true for any choice of the indices.
What do we have here? This relation tells us that A1 and A2 are independent.
This relation tells us that A1 and A3 are independent.
This relation tells us that A2 and A3 are independent.
We call this situation pairwise independence.
But the definition requires something more.
It requires that the probability of three-way intersections can also be calculated the same way by multiplying individual probabilities.
And this additional condition does make a difference, as we’re going to see in a later example.
Is this the right definition? Yes.
One can prove formally that if the conditions in this definition are satisfied, then any formula of this kind is true.
In particular, we also have relations such as the following.
The probability of event A3 is the same as the probability of event A3, given that A1 and A2 occurred.
Or the probability of A3, given that A1 occurred but A2 didn’t.
Or we can continue this similarly, the probability of A3 given that A1 did not occur, and A2 occurred, and so on.
So any kind of information that I might give you about events A1 and A2– which one of them occurred and which one didn’t– is not going to affect my beliefs about the event A3.
The conditional probabilities are going to be the same as the unconditional probabilities.
I told you that this definition implies that all relations of this kind [are] true.
This can be proved.
The proof is a bit tedious.
And we will not go through it.