In this lecture, we start our systematic study of Bayesian inference.
We will first talk a little bit about the big picture, about inference in general, the huge range of possible applications, and the different types of problems that one may encounter.
For example, we have hypothesis testing problems in which we are trying to choose between a finite and usually small number of alternative hypotheses or estimation problems where we want to estimate as close as we can an unknown numerical quantity.
We then move into the specifics of Bayesian inference.
The central idea is that we always use the Bayes rule to find the posterior distribution of an unknown random variable based on observations of a related random variable.
Depending on whether the random variables are discrete or continuous, we must of course you use the appropriate version of the Bayes rule.
If we want to summarize the posterior in a single number, that is, to come up with a numerical estimate of the unknown random variable, we then have some options.
One is to report the value at which the posterior is largest.
Another is to report the mean of the conditional distribution.
These go under the acronyms MAP and LMS.
We will see shortly what these acronyms stand for.
Given any particular method for coming up with a point estimate, there are certain performance metrics that tell us how good the estimate is.
For hypothesis testing problems, the appropriate metric is the probability of error, the probability of making a mistake.
For problems of estimating a numerical quantity, an appropriate metric that we will be using a lot is the expected value of the squared error.
As we will see, there will be no new mathematics in this lecture, just a few definitions, a few new terms, and an application of the Bayes rule.
Nevertheless, it is important to be able to apply the Bayes rule systematically and with confidence.
For this reason, we will be going over several examples.