We are now ready to move on to a model which is quite interesting and quite realistic.
This is a model in which we have an unknown parameter modeled as a random variable that we try to estimate.
This is the random variable, Theta.
And we have multiple observations of that random variable.
Each one of those observations is equal to the unknown random variable, plus some additive noise.
This is a model that appears quite often in practice.
It is often the case that we’re trying to estimate a certain quantity, but we can only observe values of that quantity in the presence of noise.
And because of the noise, what we want to do is to try to measure it to multiple times.
And so we have multiple such measurement equations.
And then we want to combine all of the observations together to come up with a good estimate of that parameter.
The assumption that we will be making are that Theta is a normal random variable.
It has a certain mean that we denote by x0.
The reason for this strange notation will be seen later.
And it also has a certain variance.
The noise terms are also normal random variables with 0 mean and a certain variance.
And finally, we assume that these basic random variables that define our model are all independent.
Based on these assumptions, now we would like to estimate Theta on the basis of the X’s.
And as usual, in the Bayesian setting, what we want to do is to calculate the posterior distribution of Theta, given the X’s.
The Bayes rule has the usual form for the case of continuous random variables.
The only remark that needs to be made is that in this case, there are multiple X’s, so X up here stands for the vector of the observations that we have obtained.
And similarly, little x will stand for the vector of the values of these observations.
So we need now to start making some progress towards calculating this term here.
What is the distribution of the vector of measurements given theta.
Before we move to the vector case, let us look at one of the measurements in isolation.
This is something that we have already seen.
If I tell you the value of the random variable, Theta, which is what happens in this conditional universe when you condition on the value of Theta, then in that universe, the random variable, Xi, is equal to the numerical value that you gave me for Theta, plus Wi.
And because Wi is independent from the random variable Theta, knowing the value of the random variable Theta does not change the distribution of Wi.
It will still have this normal distribution.
So Xi is a normal of this kind plus a constant.
And so Xi is a normal random variable with mean equal to the constant that we added, and variance equal to the original variance of the random variable, Wi.
And so we can now write down, the PDF, the conditional PDF, of Xi.
There’s going to be a normalizing constant.
And then the usual exponential term, which is going to be xi minus the mean of the distribution, which is theta.
And then we divide by the usual variance term.
Let us move next to this distribution here.
This is a shorthand notation for the joint PDF of the random variables X1 up to Xn, conditional on the random variable Theta.
So it’s really a function of multiple variables.
And how do we proceed now?
Here is the crucial observation.
If I tell you the value of the random variable capital Theta as before, then you argue as follows.
All of these random variables are independent.
So if I tell you the value of the random variable Theta, this does not change the joint distribution of the Wi’s.
The Wi’s were independent when we started, so they remain independent in the conditional universe.
And since the Wi’s are independent and Xi’s are obtained from the Wi’s by just adding a constant, this means that the Xi’s are also independent in this conditional universe.
Once I tell you the value of Theta, then because the noises are independent, the observations are also independent.
But this means that the joint PDF factors as a product of the individual marginal PDFs of the Xi’s.
And these PDFs, we have already found.
So now, we can put everything together to write down a formula for the posterior PDF using the Bayes rule.
We have this denominator term, which I will write first, and which term we do not need to evaluate.
Then we have the marginal PDF of Theta.
Now since Theta is normal with these parameters, this is of the form e to the minus theta minus x0 squared over 2 sigma 0 squared.
And then we have this joint density of the X’s conditioned on Theta, which we have already found, it is this product here.
It’s a product of n terms, one for each observation.
And each one of these terms is what we have found earlier, so I’m just substituting this expression up here.
Now once we have obtained the observations, so the value of the random variable capital X, that is, the value little x, is fixed.
Once it is fixed, then the x’s that appear here are constant.
So in particular, this term here is a constant.
We do not bother with it.
And what we have is a constant times an exponential in terms that are quadratic in theta.
So we recognize this kind of expression.
It has to correspond to a normal distribution.
And this is the first conclusion of this exercise.
That is, the posterior PDF of the parameter, Theta, given our observations, this posterior PDF is normal.
We have e to a quadratic function in theta.
And that quadratic function also involves the specific values of the X’s that we have obtained.
Let us copy what we have found and rearrange it.
Once more, we have a constant, then the exponential of the negative of some quadratic function in theta.
And the specific quadratic function that we calculated just before takes this particular form.
What is the mean of this normal distribution?
The mean is same as the peak.
And to find the peak, the location at which this PDF is largest, what we do is we try to find the place at which this quadratic function is smallest.
So what we do is to take the derivative with respect to theta of this quadratic, and set it to 0.
This gives us a sum of terms.
The derivative of the typical term is going to be theta minus xi, divided by sigma i squared.
And this expression must be equal to 0 if theta is at the peak of the posterior distribution.
And so we now rearrange this equation.
We split and take first the term involving theta, and gives us a contribution of this kind.
And we move the terms involving x’s to the other side.
And this gives us this expression.
And finally, we take this sum here and send it to the denominator of the other side.
And this gives us the final form of the solution: the peak of the posterior distribution, which is also the same as the conditional expectation of the posterior distribution.
Whenever we have a normal distribution, the expected value is the same as the place where the distribution is largest.
Let us now conclude with a few comments and words about how to interpret the result that we found.
First, let me emphasize that the same conclusions that we have obtained for the case of a single observation go through in this case as well.
The posterior distribution of the unknown parameter is still a normal distribution.
Our state of knowledge about Theta after we obtain the observations is described by a normal distribution.
Because it is a normal distribution, the location of its peak is the same as the expected value.
And for this reason, the conditional expectation estimate and the maximum a posteriori probability estimates coincide.
And finally, the form of the estimates that we get is a linear function in the xi’s.
And this linearity is a very convenient property to have, because it allows further analysis of these ways of obtaining estimates.
How do we interpret this formula?
What we have here is the following.
Each one of the xi’s gets multiplied by a certain coefficient, which is 1 over the variance.
And in the denominator, we have the sum of all of those coefficients.
So what we really have here is a weighted average of the xi’s.
Now keep in mind that those xi’s are not all of them of the same kind.
One term is x0, which is the prior mean, whereas the remaining xi’s are the values of the observations.
So there’s something interesting happening here.
We combine the values of the observations with the value of the prior mean.
And in some sense, the prior mean is treated as just one more piece of information available to us.
And it is treated in a sort of equal way as the other observations.
The weight that we have in this weighted average are that each xi gets divided by the corresponding variance.
Does this make sense?
Well, suppose that sigma i squared is large.
This means that the noise term Wi is very large.
So Xi is very noisy.
And so it’s not a useful observation to have.
And in that case, it gets a small weight.
So the weights are determined by the variances in a way that is quite sensible.
Those observations that will get the most weight will be those observations for which the corresponding noise variance is small.
So the solution to this estimation problem that we just went through has many nice properties.
First, we stay within the world of normal random variables, because even the posterior is normal.
We stay within the world of linear functions of normal random variables, and the form of the formula that we have, itself, has an appealing intuitive content.