6.041 / 6.431 21.Bayesian Statistical Inference - I

� LECTURE 21 • Readings: Sections 8.1-8.2 “It is the mark of truly educated people to be deeply moved by statistics.” (Oscar Wilde) Reality (e.g., customer arrivals) Model (e.g., Poisson) Data • Design & interpretation of experiments – polling, medical/pharmaceutical trials... • Netflix competition • Finance 2 1 4 5 4 1 3 5 5 4 1 3 3 5 2 4 5 3 2 1 4 1 5 5 4 2 5 4 3 3 1 5 2 1 3 1 2 3 4 5 1 3 3 3 5 2 1 1 5 2 4 4 1 3 1 5 4 5 1 2 4 5 ? ? ? ? ? ? ? ? ? ? • Signal processing – Tracking, detection, speaker identification,... Types of Inference models/approaches • Model building versus inferring unknown variables. E.g., assume X = aS + W – Model building: know “signal” S, observe X, infer a – Estimation in the presence of noise: know a, observe X, estimate S. • Hypothesis testing: unknown takes one of few possible values; aim at small probability of incorrect decision • Estimation: aim at a small estimation error • Classical statistics: Estimator ˆΘX N pX(x; θ)θ θ: unknown parameter (not a r.v.) mass of electron •Use priors & Bayes rule Estimator ˆΘX pX|Θ(x | θ) N pΘ(θ) Θ Graph of S&P 500 index removed due to copyright restrictions. ◦ E.g., θ = Bayesian: Bayesian inference: Use Bayes rule • Hypothesis testing – discrete data pΘ|X(θ | x)= pΘ(θ) pX|Θ(x | θ) pX(x) – continuous data pΘ|X(θ | x)= pΘ(θ) fX|Θ(x | θ) fX(x) • Estimation; continuous data fΘ|X(θ | x)= fΘ(θ) fX|Θ(x | θ) fX(x) Zt =Θ0 + tΘ1 + t2Θ2 Xt = Zt + Wt, t =1,2,...,n Bayes rule gives: fΘ0,Θ1,Θ2|X1,...,Xn(θ0,θ1,θ2 | x1,...,xn) Estimation with discrete data fΘ(θ) pX|Θ(x | θ) fΘ|X(θ | x)= pX(x) pX(x)= fΘ(θ)pX|Θ(x | θ) dθ • Example: – Coin with unknown parameter θ – Observe X heads in n tosses • What is the Bayesian approach? – Want to find fΘ|X(θ | x) – Assume a prior on Θ (e.g., uniform) Output of Bayesian Inference • Posterior distribution: – pmf pΘ|X(·|x)orpdf fΘ|X(·|x) • If interested in a single answer: – Maximum a posteriori probability (MAP): ◦ pΘ|X(θ∗|x) = maxθpΘ|X(θ |x) minimizes probability of error; often used in hypothesis testing ◦ fΘ|X(θ ∗|x) = maxθfΘ|X(θ |x) – Conditional expectation: E[Θ |X = y]= � θfΘ|X(θ |x) dθ – Single answers can be misleading! Least Mean Squares Estimation • Estimation in the absence of information 1/6 4 10 θ fΘ(θ) • find estimate c, to: minimize E � (Θ −c)2 � • Optimal estimate: c = E[Θ] • Optimal mean squared error: E � (Θ −E[Θ])2 � =Var(Θ) LMS Estimation of Θ based on X • Two r.v.’s Θ, X • we observe that X = x – new universe: condition on X = x • E � (Θ −c)2 |X = x � is minimized by c = • E � (Θ −E[Θ |X = x])2 |X = x � ≤E[(Θ −g(x))2 |X = x] ◦E � (Θ −E[Θ |X])2 |X � ≤E � (Θ −g(X))2 |X � ◦ E � (Θ −E[Θ |X])2 � ≤E � (Θ −g(X))2 � E[Θ |X] minimizes E � (Θ −g(X))2 � over all estimators g(·) LMS Estimation w. several measurements • Unknown r.v. Θ • Observe values of r.v.’s X1,...,Xn • Best estimator: E[Θ |X1,...,Xn] • Can be hard to compute/implement – involves multi-dimensional integrals, etc. MIT OpenCourseWare http://ocw.mit.edu 6.041 /6.431 Probabilistic Systems Analysis and Applied Probability Fall 2010 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.