We will now continue with the problem of inferring the unknown bias of a certain coin for which we have a certain prior distribution and of which we observe the number of heads in n independent coin tosses.
We have already seen that if we assume a uniform prior, the posterior takes this particular form, which comes from the family of Beta distributions.
What we want to do now is to actually derive point estimates.
That is, instead of just providing the posterior, we would like to select a specific estimate for the unknown bias.
Let us look at the maximum a posteriori probability estimate.
How can we find it?
By definition, the MAP estimate is that value of theta that maximizes the posterior, the value of theta at which the posterior is largest.
Now, instead of maximizing the posterior, it is more convenient in this example to maximize the logarithm of the posterior.
And the logarithm is k times log theta, plus n minus k times the log of 1 minus theta.
To carry out the maximization over theta, we form the derivative with respect to theta and set that derivative to 0.
So the derivative of the first term is k over theta.
And the derivative of the second term is n minus k over this quantity, 1 minus theta.
But because of the minus here when we apply the chain rule, actually, this plus sign here is going to become a minus sign.
And now we set this derivative to 0.
We carry out the algebra, which is rather simple.
And the end result that you will find is that the estimate is equal to k over n.
Notice that this is lowercase k.
We are told the specific value of heads that has been observed.
So little k is a number, and our estimate, accordingly, is a number.
This answer makes perfect sense.
A very reasonable way of estimating the probability of heads of a certain coin is to look at the number of heads obtained and divide by the total number of trials.
So we see that the MAP estimate turns out to be a quite natural one.
How about the corresponding estimator?
Recall the distinction that the estimator is a random variable that tells us what the estimate is going to be as a function of the random variable that is going to be observed.
The estimator is uppercase K divided by little n.
So it is a random variable whose value is determined by the value of the random variable capital K.
If the random variable capital K happens to take on this specific value, little k, then our estimator, this random variable, will be taking this specific value, which is the estimate.
And let us now compare with an alternative way of estimating Theta.
We will consider estimating Theta by forming the conditional expectation of Theta, given the specific number of heads that we have observed.
This is what we call the LMS or least mean squares estimate.
To calculate this conditional expectation, all that we need to do is to form the integral of theta times the density of Theta.
But since it’s a conditional expectation, we need to take the conditional density of Theta.
And the integral ranges from 0 to 1, because this is the range of our random variable, Theta.
Now, what is this?
We have a formula for the posterior density.
So we need to just multiply this expression here by theta, and then integrate.
This term here is a constant.
So it can be pulled outside the integral.
And inside the integral, we are left with this term times theta, which changes the exponent of theta to k plus 1.
Then we have 1 minus theta to the power n minus k, d theta.
At this point, we need to do some calculations.
What is d of n, k?
d of n, k is the normalizing constant of this PDF.
For this to be a PDF and to integrate to 1, d of n, k has to be equal to the integral of this expression from 0 to 1.
So we need to somehow be able to evaluate this integral.
Here, we will be helped by the following very nice formula.
This formula tells us that the integral of for such a function of theta from 0 to 1 is equal to this very nice and simple expression.
Of course, this formula is only valid when these factorials make sense.
So we assume that alpha is non-negative and theta is non-negative.
How is this formula derived?
There’s various algebraic or calculus style derivations.
One possibility is to use integration by parts.
And there are also other tricks for deriving it.
It turns out that there is also a very clever probabilistic proof of this fact.
But in any case, we will not derive it.
We will just take it as a fact that comes to us from calculus.
And now, let us apply this formula.
d of n, k is equal to the integral of this expression, which is of this form, with alpha equal to k and beta equal to n minus k.
So d of n, k takes the form alpha is k, beta is n minus k.
And then in the denominator, we have the sum of the two indices plus 1.
So it’s going to be k plus n minus k.
That gives us n.
And then there’s a plus 1.
And how about this integral?
Well, this integral is also of the form that we have up here.
But now, we have alpha equal to k plus 1, beta is n minus k.
And in the denominator, we have the sum of the indices plus 1.
So when we add these indices, we get n plus 1.
And then we get another factor of 1, which gives us an n plus 2.
This looks formidable.
But actually, there’s a lot of simplifications.
This term here cancels with that term.
k plus 1 factorial divided by k factorial, what is it?
It is just a factor of k plus 1.
And what do we have here?
This term is in the denominator of the denominator.
So it can be moved up to the numerator.
We have n plus 1 factorial divided by n plus 2 factorial.
This is just n plus 2.
And this is the final form of the answer.
This is what the conditional expectation of theta is.
So now, we can compare the two estimates that we have, the MAP estimate and the conditional expectation estimate.
They’re fairly similar, but not exactly the same.
This means that the mean of a Beta distribution is not the same as the point at which the distribution is highest.
On the other hand, if n is a very large number, this expression is going to be approximately equal to k over n when n is large.
And so in the limit of large n, the two estimators will not be very different from each other.