Let us now go through an example.
Suppose that we have an unknown random variable Theta that has a uniform distribution between 4 and 10.
We observe some other random variable X that’s related to Theta according to the following model.
This is the conditional distribution of X given Theta.
For any given value of theta, X is going to take values between theta minus 1 and theta plus 1.
And the conditional distribution is uniform on that range.
One way of thinking about this particular observation model is that what we observe is the true value of Theta plus some noise term.
And this noise term is uniform on the range from minus 1 to plus 1.
So given a value of Theta, we may observe anything, because of noise, that’s up to one lower or one higher than the true value of Theta.
And if we take this description, actually, this random variable U has this distribution no matter what Theta is.
And therefore, U is independent of Theta.
But in any case, this particular interpretation will not matter.
Let us see how do we proceed.
In Bayesian estimation, the first step is always to put our hands on the posterior distribution of Theta.
And to find the posterior, we can start by first finding the joint.
So let us look at the x theta plane.
That’s where the joint distribution is going to live.
And our first step will be to locate those values of X and Theta that are possible, given our description.
From this model here, we know that theta minus 1 is going to be less than or equal to x.
And x is going to be less than or equal to theta plus 1.
And we translate this into two inequalities, namely that theta is less than or equal to x plus 1, and from here, that theta is larger than or equal to x minus 1.
So these are the constraints that we have on the possible values of x and theta.
So here we plot the line where theta is equal to x plus one.
And here we plot the line on which theta is equal to x minus 1.
And these two inequalities that we’ve got here tell us that we need to live somewhere in between those two lines.
In addition, we have the fact that theta lives between 4 and 10.
And that places these two limits as well.
So to summarize, this shape here is the set off all possible x’s and thetas.
Outside this shape, the joint PDF is going to be zero.
What is it going to be inside here?
Well, because the prior is uniform, that is, it is constant, and the model is also uniform, to obtain the joint we multiply these two.
And since they are constant, we obtain a joint that’s also constant.
So the joint PDF is equal to a constant over that set.
We can easily calculate the area of this set.
It is 12.
So the joint is equal to 1 over 12 inside this set.
And of course, it’s 0 elsewhere.
So we have a uniform joint PDF.
Now, let us look at the posterior.
If I tell you that X takes on this specific value, this means that we now live in this universe.
And it means that all of those thetas are possible.
The posterior distribution is going to be a distribution that tells us the probabilities of these different thetas.
What kind of distribution is it?
Well, we know that the conditional is just a section out of the joint but keeps, otherwise, the same shape.
Since the joint is constant, it’s uniform over that set, it means that the posterior, or the conditional, is also constant over that set.
So the conclusion is that the posterior distribution of Theta is a uniform distribution on this set.
Given this knowledge, what is the conditional expectation?
The conditional expectation of a uniform is just the midpoint of that uniform.
And so this is going to be our estimate of Theta, the conditional expectation of Theta, given the observation that we have obtained.
And then a similar argument applies no matter what x we have obtained.
For any given x, our estimate is going to be the midpoint of the corresponding interval.
So what kind of shape do we get by doing this, by joining the mid-points?
It’s going to be a straight line over this region.
It’s also going to be a straight line over this region except that, because of the change in shape, it’s going to be a straight line with a different slope.
And similarly, in this region, it’s also going to be a straight line with a different slope.
So what have we plotted here?
For any given value of X, we have plotted the corresponding conditional expectation of Theta given that value of X.
And as a function of x, this gives us a certain curve.
And this blue curve that we have calculated is a particular function of x.
And we can think of this function g as being our estimator.
So the way we’re going to be processing the data will be that whenever we obtain an x, we apply this particular function g.
And we come up with an estimate.