We have so far discussed the first step involved in the construction of a probabilistic model. Namely, the construction of a sample space, which is a description of the possible outcomes of a probabilistic experiment.
We now come to the second and much more interesting part. We need to specify which outcomes are more likely to occur and which ones are less likely to occur and so on. And we will do that by assigning probabilities to the different outcomes. However, as we try to do this assignment, we run into some kind of difficulty, which is the following.
Remember the previous experiment involving a continuous sample space, which was the unit square and in which we throw a dart at random and record the point that occurred. In this experiment, what do you think is the probability of a particular point? Let’s say what is the probability that my dart hits exactly the center of this square. Well, this probability would be essentially 0.
Hitting the center exactly with infinite precision should be 0. And so it’s natural that in such a continuous model any individual point should have a 0 probability. For this reason instead of assigning probabilities to individual points, we will instead assign probabilities to whole sets, that is, to subsets of the sample space. So here we have our sample space, which is some abstract set omega.
Here is a subset of the sample space. Call it capital A. We’re going to assign a probability to that subset A, which we’re going to denote with this notation, which we read as the probability of set A. So probabilities will be assigned to subsets.
And these will not cause us difficulties in the continuous case because even though individual points would have 0 probability, if you ask me what are the odds that my dart falls in the upper half, let’s say, of this diagram, then that should be a reasonable positive number.
So even though individual outcomes may have 0 probabilities, sets of outcomes in general would be expected to have positive probabilities. So coming back, we’re going to assign probabilities to the various subsets of the sample space. And here comes a piece of terminology, that a subset of the sample space is called an event.
Why is it called an event? Because once we carry out the experiment and we observe the outcome of the experiment, either this outcome is inside the set A and in that case we say that event A has occurred, or the outcome falls outside the set A in which case we say that event A did not occur.
Now we want to move on and describe certain rules. The rules of the game in probabilistic models, which are basically the rules that these probabilities should satisfy. They shouldn’t be completely arbitrary.
First, by convention, probabilities are always given in the range between 0 and 1. Intuitively, 0 probability means that we believe that something practically cannot happen. And probability of 1 means that we’re practically certain that an event of interest is going to happen.
So we want to specify rules of these kind for probabilities. These rules that any probabilistic model should satisfy are called the axioms of probability theory.
And our first axiom is a nonnegativity axiom. Namely, probabilities will always be non-negative numbers. It’s a reasonable rule.
The second rule is that if the subset that we’re looking at is actually not a subset but is the entire sample space omega, the probability of it should always be equal to 1. What does that mean? We know that the outcome is going to be an element of the sample space.
This is the definition of the sample space.
So we have absolute certainty that our outcome is going to be in omega. Or in different language we have absolute certainty that event omega is going to occur. And we capture this certainty by saying that the probability of event omega is equal to 1.
These two axioms are pretty simple and very intuitive. The more interesting axiom is the next one that says something a little more complicated. Before we discuss that particular axiom, a quick reminder about set theoretic notation.
If we have two sets, let’s say a set A, and another set, another set B, we use this particular notation, which we read as “A intersection B” to refer to the collection of elements that belong to both A and B. So in this picture, the intersection of A and B is this shaded set.
We use this notation, which we read as “A union B”, to refer to the set of elements that belong to A or to B or to both. So in terms of this picture, the union of the two sets would be this blue set.
After this reminder about set theoretic notation, now let us look at the form of the third axiom. What does it say? If we have two sets, two events, two subsets of the sample space, which are disjoint.
So here’s our sample space. And here are the two sets that are disjoint. In mathematical terms, two sets being disjoint means that their intersection has no elements. So their intersection is the empty set.
And we use this symbol here to denote the empty set. So if the intersection of two sets is empty, then the probability that the outcome of the experiments falls in the union of A and B, that is, the probability that the outcome is here or there, is equal to the sum of the probabilities of these two sets. This is called the additivity axiom.
So it says that we can add probabilities of different sets when those two sets are disjoint. In some sense we can think of probability as being one pound of some substance which is spread over our sample space and the probability of A is how much of that substance is sitting on top of a set A.
So what this axiom is saying is that the total amount of that substance sitting on top of A and B is how much is sitting on top of A plus how much is sitting on top of B. And that is the case whenever the sets A and B are disjoint from each other.
The additivity axiom needs to be refined a bit. We will talk about that a little later. Other than this refinement, these three axioms are the only requirements in order to have a legitimate probability model. At this point you may ask, shouldn’t there be more requirements?
Shouldn’t we, for example, say that probabilities cannot be greater than 1?
Yes and no. We do not want probabilities to be larger than 1, but we do not need to say it. As we will see in the next segment, such a requirement follows from what we have already said.
And the same is true for several other natural properties of probabilities.