In this segment we start a new topic.
We will talk about the covariance of two random variables, which gives us useful information about the dependencies between these two random variables.
Let us motivate the concept by looking first at a special case.
Suppose that X and Y have zero means and that they there are discrete random variables.
If X and Y are independent, then the expectation of the product is the product of the expectations.
And since we have assumed zero means, this is going to be equal to zero.
But suppose instead that the joint PMF of X and Y is of the following kind.
Each point in this diagram is equally likely, so we have here a discrete uniform distribution on the discrete set which consists of the points shown in this diagram.
What we have in this particular example is that at most outcomes, positive values of X tend to go together with positive values of Y.
And negative values of X tend to go together with negative values of Y.
So most of the time we have outcomes in this quadrant, in which x times y is positive, or in this quadrant where x times y is, again, positive.
But some of the time we fall in this quadrant where x times y is negative, or in this quadrant where x times y is negative.
Since we have many more points here and here, on the average, the value of x times y is going to be positive.
On the other hand, if the diagram takes this form, then, most of the time, the pair x, y lies in this quadrant or in that quadrant where the product of x times y is negative.
So the random variables X and Y typically have opposite signs, and on the average, the expected value of X times Y is going to be negative.
So here we have a positive expectation, here we have a negative expectation of X times Y.
This quantity, the expected value of X times Y, tells us whether X and Y tend to move in the same or in opposite directions.
And this quantity is what we call the covariance, in the zero mean case.
Let us now generalize.
The random variables do not have to be discrete.
This quantity is well defined for any kind of random variables.
And if we have non-zero means, the covariance is defined by this expression.
What we have here is that we look at the deviation of X from its mean value, and the deviation of Y from its mean value, and we’re asking whether these two deviations tend to have the same sign or not, whether they move in the same direction or not.
If the covariance is positive, what it tells us is that whenever this quantity is positive so that X is above its mean, then, typically or usually, the deviation of Y from its mean will also tend to be positive.
To summarize, the covariance, in general, tells us whether two random variables tend to move together, both being high or both being low, in some average or typical sense.
Now, if the two random variables are independent, we already saw that in the zero mean case, this quantity– the covariance– is going to be 0.
How about the case where we have non-zero means?
Well, if we have independence, then we have the expected value of the product of two random variables.
X and Y are independent, so X minus the expected value, which is a constant, is going to be independent from Y minus its expected value.
And so, the covariance is going to be the product of two expectations.
But the expected value of X minus this constant is 0, and the same is true for this term as well.
So the covariance in this case is going to be equal to 0.
So in the independent case, we have zero covariances.
On the other hand, the converse is not true.
There are examples in which we have dependence but zero covariance.
Here is one example.
In this example there are four possible outcomes.
At any particular outcome, either X or Y is going to be 0.
So in this example the random variable X times Y is identically equal to 0.
The mean of X is also 0, the mean of Y is also 0 by symmetry, so the covariance is the expected value of this quantity.
And so the covariance, in this example, is equal to 0.
On the other hand, the two random variables, X and Y, are dependent.
If I tell you that X is equal to 1, then you know that this outcome has occurred.
And in that case, you are certain that Y is equal to 0.
So knowing the value of X tells you a lot about the value of Y and, therefore, we have dependence between these two random variables.