In this lecture, we develop the weak law of large numbers. Loosely speaking, the weak law of large numbers says that if we have a sequence of independent random variables with the same distribution, then the average of these random variables, which is called the sample mean, approaches the expected value of the distribution. In …
Introduction. In this final segment, we want to discuss an interesting point about linear estimators. Here’s what the issue is. You obtain an observation, X, on the basis of which you want to estimate Theta. But perhaps you measure X on a different scale, let’s say on a cubic scale, so that what you record …
Probability – L17.9 The Representation of the Data Matters in LLMS Read More »
Introduction. As we discussed earlier– even if we have multiple observations we can still find the structure of the best linear estimator in a fairly simple, computational way– by solving a system of linear equations. But usually, we do not get nice and simple formulas. But here is a nice example, in which we will …
Probability – L17.8 The Simplest LLMS Example with Multiple Observations Read More »
We have presented the complete solution to the liner least mean squares estimation problem, when we want to estimate a certain unknown random variable on the basis of a different random variable X that we get to observe. But what if we have multiple observations? What would be the analogous formulation of the problem? Here’s …
Probability – L17.7 LLMS with Multiple Observations Read More »
Let’s now go through another example, which will be a little more challenging. We’re going to revisit an old problem. We have a coin that has an unknown bias, Theta. And we have a prior distribution on this Theta. We fix some positive integer, n, we flip a coin n times, that has this unknown …
Probability – L17.6 LLMS for Inferring the Parameter of a Coin Read More »
Intro. Let us now illustrate the linear least mean squares estimation methodology in the context of an example. And we’re going to revisit our familiar example that we considered earlier in the context of general least mean squares estimation. Assumptions. Let us remind ourselves what were the assumptions behind this example. There is an unknown …
Now that we have found the solution to the linear least mean squares estimation problem, it is time to offer a few comments, make some observations, and provide some insights. A first important observation is the following. In order to implement this estimator, you do not really need to know everything about the distribution of …
Probability – L17.4 Remarks on the LLMS Solution and on the Error Variance Read More »
We will now develop the solution to the linear least mean squares estimation problem. We want to find coefficients a and b that make this expression, this mean squared error, as small as possible. We will approach this problem by proceeding in two stages. In the first stage, we assume that the choice of a …
Probability – L17.3 Solution to the LLMS Problem Read More »
Let us now introduce the linear least mean squares formulation. The setting is the usual one– we have an unknown random variable and another random variable, which is our observation. We’re given enough information so that we can, for example, calculate the joint distribution of these two random variables. What we would like to do …
If our objective is to keep the mean squared estimation error small, then the best possible estimator is the conditional expectation. But sometimes the conditional expectation is hard to calculate. Maybe we’re missing the details of the various probability distributions. Or maybe we have the distributions that we need but the formulas are complicated. After …