In the previous example, we had a model where the result of the first coin toss did not affect the probabilities of what might happen in the second toss.
This is a phenomenon that we call independence and which we now proceed to define.
Let us start with a first attempt at the definition.
We have an event, B, that has a certain probability of occurring.
We are then told that event A occurred, but suppose that this knowledge does not affect our beliefs about B in the sense that the conditional probability remains the same as the original unconditional probability.
Thus, the occurrence of A provides no new information about B.
In such a case, we may say that event B is independent from event A.
If this is indeed the case, notice that the probability that both A and B occur, which is always equal by the multiplication rule to the probability of A times the conditional probability of B given A.
So this is a relation that’s always true.
But if we also have this additional condition, then this simplifies to the probability of A times the probability of B.
So we can find the probability of both events happening by just multiplying their individual probabilities.
It turns out that this relation is a cleaner way of the defining formally the notion of independence.
So we will say that two events, A and B, are independent if this relation holds.
Why do we use this definition rather than the original one? This formal definition has several advantages.
First, it is consistent with the earlier definition.
If this equality is true, then the conditional probability of event B given A, which is the ratio of this divided by that, will be equal to the probability of B.
So if this relation holds, then this relation will also hold, and so this more formal definition is consistent with our earlier intuitive definition.
A more important reason is that this formal definition is symmetric with respect to the roles of A and B.
So instead of saying that B is independent from A, based on this definition we can now say that events A and B are independent of each other.
And in addition, since this definition is symmetric and since it implies this condition, it must also imply the symmetrical relation.
Namely, that the conditional probability of A given B is the same as the unconditional probability of A.
Finally, on the technical side, conditional probabilities are only defined when the conditioning event has non-zero probability.
So this original definition would only make sense in those cases where the probability of the event A would be non-zero.
In contrast, this new definition makes sense even when we’re dealing with zero probability events.
So this definition is indeed more general, and this also makes it more elegant.
Let us now build some understanding of what independence really is.
Suppose that we have two events, A and B, both of which have positive probability.
And furthermore, these two events are disjoint.
They do not have any common elements.
Are these two events independent? Let us check the definition.
The probability that both A and B occur is zero because the two events are disjoint.
They cannot happen together.
On the other hand, the probability of A times the probability of B is positive, since each one of the two terms is positive.
And therefore, these two expressions are different from each other, and therefore this equality that’s required by the definition of independence does not hold.
The conclusion is that these two events are not independent.
In fact, intuitively, these two events are as dependent as Siamese twins.
If you know that A occurred, then you are sure that B did not occur.
So the occurrence of A tells you a lot about the occurrence or non-occurrence of B.
So we see that being independent is something completely different from being disjoint.
Independence is a relation about information.
It is important to always keep in mind the intuitive meaning of independence.
Two events are independent if the occurrence of one event does not change our beliefs about the other.
It does not affect the probability that the other event also occurs.
When do we have independence in the real world? The typical case is when the occurrence or non-occurrence of each of the two events A and B is determined by two physically distinct and non-interacting processes.
For example, whether my coin results in heads and whether it will be snowing on New Year’s Day are two events that should be modeled as independent.
But I should also say that there are some cases where independence is less obvious and where it happens through a numerical accident.
You can now move on to answer some simple questions where you will have to check for independence using either the mathematical or intuitive definition.