Modeling and Inference: A Big Picture

Suppose we are interested in studying a certain phenomenon that can be schematically represented as follows: it is a “black box” that takes inputs and produces responses. Furthermore, suppose we collected some data — a set of inputs and the corresponding responses — by observation or experiment. What can we learn, or infer, about the phenomenon from the data? There are two fundamental goals of data analysis: 1) Understanding: We want to understand the mechanism of how the responses are associated with the inputs. What is “inside” the Nature “black box”? 2) Prediction: We want to learn how to predict the response to a future input. The central idea of statistical inference is to replace the Nature “black box”, i.e. the unknown mechanism that Nature uses to associate the responses with the inputs, with a stochastic model. The key feature of a stochastic model is that the observed variability in the data is modeled by probability distributions, which form the building blocks of the model. In other words, the data is viewed as a collection of realizations of random variables. In this chapter, we will define the two main types of stochastic models and provide a big picture of several fundamental concepts and problems of statistical inference.