Let us now revisit the subject of expectations and develop an important linearity property for the case where we’re dealing with multiple random variables.
We already have a linearity property.
If we have a linear function of a single random variable, then expectations behave in a linear fashion.
But now, if we have multiple random variables, we have this additional property.
The expected value of the sum of two random variables is equal to the sum of their expectations.
Let us go through the derivation of this very important fact because it is a nice exercise in applying the expected value rule and also manipulating PMFs and joint PMFs.
We’re dealing with the expected value of a function of two random variables.
Which function? If we write it this way, we are dealing with the function g, which is just the sum of its two entries.
So now we can continue with the application of the expected value rule.
And we obtain the sum over all possible x, y pairs.
Here, we need to write to g of x,y.
But in our case, the function we’re dealing with is just x plus y.
And then we weigh, according to the entries of the joint PMF.
So this is just an application of the expected value rule to this particular function.
Now let us take this sum and break it into two pieces– one involving only the x-term, and another piece involving only the y-term.
Now, if we look at this double summation, look at the inner sum.
It’s a sum over y’s.
While we’re adding over y’s, the value of x remains fixed.
So x is a constant, as far as the sum is concerned.
So x can be pulled outside this summation.
Let us just continue with this term, the first one, and see that a simplification happens.
This quantity here is the sum of the probabilities of the different y’s that can go together with a particular x.
So it is just equal to the probability or that particular x.
It’s the marginal PMF.
If we carry out a similar step for the second term, we will obtain the sum over y’s.
It’s just a symmetrical argument.
And at this point we recognize that what we have in front of us is just the expected value of X, this is the first term, plus the expected value of Y.
So this completes the derivation of this important linearity property.
Of course, we proved the linearity property for the case of the sum of two random variables.
But you can proceed in a similar way, or maybe use induction, and one can easily establish, by following the same kind of argument, that we have a linearity property when we add any finite number of random variables.
The expected value of a sum is the sum of the expected values.
Just for a little bit of practice, if, for example, we’re dealing with this expression, the expected value of that expression would be the expected value of 2X plus the expected value of 3Y minus the expected value of Z.
And then, using the linearity property for linear functions of a single random variable, we can pull the constants out of the expectations.
And this would be twice the expected value of X plus 3 times the expected value of Y minus the expected value of Z.
What we will do next is to use the linearity property of expectations to solve a problem that would otherwise be quite difficult.
We will use the linearity property to find the mean of a binomial random variable.
Let X be a binomial random variable with parameters n and p.
And we can interpret X as the number of successes in n independent trials where each one of the trials has a probability p of resulting in a success.
Well, we know the PMF of a binomial.
And we can use the definition of expectation to obtain this expression.
This is just the PMF of the binomial.
And therefore, what we have here is the usual definition of the expected value.
Now, if you look at this sum, it appears quite formidable.
And it would be quite hard to evaluate it.
Instead, we’re going to use a very useful trick.
We will employ what we have called indicator variables.
So let’s define a random variable Xi, which is a one if the ith trial is a success, and zero otherwise.
Now if we want to count successes, what we want to count is how many of the Xi’s are equal to 1.
So if we add the Xi’s, this will have a contribution of 1 from each one of the successes.
So when you add them up, you obtain the total number of successes.
So we have expressed a random variable as a sum of much simpler random variables.
So at this point, we can now use linearity of expectations to write that the expected value of X will be the expected value of X1 plus all the way to the expected value of Xn.
Now what is the expected value of X1? It is a Bernoulli random variable that takes the value 1 with probability p and takes the value of 0 with probability 1 minus p.
The expected value of this random variable is p.
And similarly, for each one of these terms in the summation.
And so the final end result is equal to n times p.
This answer, of course, makes also intuitive sense.
If we have to p equal to 1/2, and we toss a coin 100 times, the expected number, or the average number, of heads we expect to see will be 1/2 half times 100, which is 50.
The higher p is, the more successes we expect to see.
And of course, if we double n, we expect to see twice as many successes.
So this is an illustration of the power of breaking up problems into simpler pieces that are easier to analyze.
And the linearity of expectations is one more tool that we have in our hands for breaking up perhaps complicated random variables into simpler ones and then analyzing them separately.