In this segment, we’re going to go over a few theoretical properties of the estimation error in least mean squares estimation.
Recall that our least mean squares estimator is the conditional expectation of the unknown random variable, given our observations.
Let us define the error, which is the difference between the estimator and the random variable that we are trying to estimate.
Let us start with some observations.
What is the expected value of our estimator?
Well, using the law of iterated expectations, the expectation of a conditional expectation is the same as the unconditional expectation.
And using this property, by moving this Theta to the other side, what we obtain is that the estimation error has an expectation of 0.
So this tells us that the estimation error, on the average, is equal to 0, which is good news.
In fact, something stronger is true.
Not just the overall average of the estimation error is 0, but even if you condition on a particular measurement, still the conditional expectation of your estimation error is going to be equal to 0.
Let us derive this relation.
We’re looking at the expected value of Theta tilde, which is Theta hat minus Theta, conditional on a value of X.
Now, if I tell you the value of X, then the estimator is completely determined– there’s no uncertainty about it– so the expectation of Theta hat, in this conditional universe, is just Theta hat itself.
And we’re left with the second term, but the second term is also Theta hat, and therefore we obtain a difference of 0.
Let us now move to a slightly more complicated question.
What is the covariance between the estimation error and the estimate?
We will calculate the covariance as follows.
It is the expected value of the product of the two random variables that we are interested in, minus the product of their expectations.
Now, we already calculated that the expected value of the estimation error is equal to 0, and therefore, this term here disappears.
This term is equal to 0.
So we now need to calculate the first term.
This may seem difficult, but conditioning is always a great trick, so let’s do that.
Let us start by calculating the conditional expectation of this product.
As before, in the conditional universe, where we’re told the value of X, the value of Theta hat is known.
It is becoming a constant, so it can be pulled outside the expectation.
But then we can apply the fact that we established earlier that this term is 0, and therefore, we obtain a 0 here.
Now, the expected value of a random variable is the same as the expected value of the conditional expectation.
This is, again, the law of iterated expectations.
Since the conditional expectation is 0, when we apply the law of iterated expectations to this quantity, we also obtain a 0.
Therefore, this term is 0 as well, and we have established what we wanted to show.
Using this fact, now we can figure out that the following is true.
We write the random variable Theta as the sum of Theta hat minus Theta tilde.
This comes simply from this definition here, by just moving Theta to this side, and Theta tilde to the other side.
So Theta is the difference of two random variables, and these two random variables have 0 covariance.
When two random variables have 0 covariance, then the variance of their sum, or of their difference, is the sum of the variances.
And this leads us to this relation– that the variance of our random variable can be decomposed into two pieces.
One of them is the variance of the estimator, and the other is the variance of the estimation error.
This is an interesting fact.
It can actually be derived in a different way, as well.
It is just a manifestation of the law of total variances, but hidden in somewhat different notation.
And this concludes our discussion of theoretical properties of the estimation error.
Unfortunately we will not have the opportunity to use them in any interesting ways.
On the other hand, they are a foundational piece for the more general theory of least-squares estimation.
If you try to develop it in a more sophisticated and more deep way, it turns out that these properties are cornerstones of that theory.