In the next variation that we consider, the random variable Theta is still discrete.
So it might, for example, represent a number of alternative hypothesis.
But now our observation is continuous.
Of course, we do have a variation of the Bayes rule that’s applicable to this situation.
The only difference from the previous version of the Bayes rule is that now the PMF of X, the unconditional and the conditional one, is replaced by a PDF.
Otherwise, everything remains the same.
A standard example is the following.
Here we’re sending a signal that takes one of, let’s say, three alternative values.
And what we observe is the signal that was sent plus some noise.
And the typical assumption here might be that the noise has zero mean and a certain variance, and is independent from the signal that was sent.
This is an example that we more or less studied some time ago.
Actually, at that time, we looked at an example where Theta could only take one out of two values, but the calculations and the methodology remains essentially the same as for the case of three values.
So in principle, we do know at this point how to apply the Bayes rule in this situation to come up with a conditional PMF of theta.
And the key to that calculation was that the term that we need, the conditional PDF of X, can be obtained from this equation as follows.
If I tell you the value of Theta, then X is essentially the same as W plus a certain constant.
Adding a constant just shifts the PDF of W by an amount equal to that constant.
And, therefore, the conditional PDF of X is the shifted PDF of the random variable W.
Using this particular fact, we can then apply the Bayes rule, carry out of the calculations, and suppose that in the end we came up with these results.
That is we obtain the specific observation x and based on that observation, we calculate the conditional probabilities of the different choices of Theta.
At this point, we may use the MAP rule and come up with an estimate which is the value of Theta, which is the more likely one.
And then we can continue exactly as in the case of discrete measurements, of discrete observations, and talk about conditional probabilities of error and so on.
Now, the fact that X is continuous really makes no difference, once we arrive at this picture.
With the MAP rule we still choose the most likely value of theta, and this is our estimates.
And we can calculate the probability of error, which with the MAP rule would be 0.
4, exactly the same argument as for the case of discrete observations applies and shows that this conditional probability of error is smallest under the MAP rule.
And then we can continue similarly and talk about the overall probability of error, which can be calculated using the total probability theorem in two ways.
One way is to take the conditional probability of error for any given value of X and then average those conditional probabilities of errors over all the possible choices of X.
Because X is now continuous, here we’re going to have an integral.
Alternatively, you can condition on the possible values of Theta, calculate conditional probabilities of error for any particular choice of theta, and then take a weighted average of them.
In practice, this calculation sometimes turns out to be the simpler one.
Finally, we can replicate the argument that we had in the discrete case.
Since the MAP rule makes this term here as small as possible, it is less than or equal to the probability of error that you would get under any other estimate or estimator, then it follows that the integral will also be as small as possible.
And therefore, the conclusion is that the overall probability of error is, again, the smallest possible when we use the MAP rule.
And so the MAP rule remains the optimal way of choosing between alternative hypothesis, whether X is discrete or continuous.