Approximations to Bayes

Bayes’ Theorem tells us how to learn from data. We just need to assign our prior probability distribution for parameters or models and formulate the data likelihood function. The posterior distribution is then fully determined, and it encapsulates everything of interest. With the posterior in hand, we can make predictions with proper uncertainty quantification, we can carry out risk analysis and provide decision support. So the basic ideas are extremely simple and powerful. But we have also seen that assigning a prior and likelihood is not always easy, and deriving a sample from the posterior distribution may require computationally demanding methods such as MCMC. So people keep searching for shortcuts where the Bayesian analysis can be made faster albeit perhaps a little bit less informative and accurate. Martin et al. (Stat Sci 39:20–45, 2023) provide a good overview of these various attempts. Some of the approximation methods date from times when computers were slow and application of sampling-based Bayesian analysis was inevitably time-consuming. But the problem with computational demand has not gone away. The advent of complex computer simulators, such as global climate models (GCM) and dynamic vegetation models (DGVM), is keeping computational efficiency high on the agenda. There will always remain an important role in computational statistics for approximation methods. But I do believe that they are over-used. There is no better way of assessing your own understanding of a system than thinking carefully about all the system’s parameters and about all measurements and their uncertainties. Writing down a prior and likelihood function is a learning experience in itself. And the posterior gives you more information than any of the outcomes from approximation can do. Another reason to prefer the real Bayesian deal over approximation methods is that Bayes’ Theorem makes no assumptions about how many data you have (you don’t need infinitely many) or about your parameter uncertainties (they need not be uniform or Gaussian) or your model (it need not be linear in any way). Bayes gives you the freedom to express exactly what you know and don’t know. Bayes’ Theorem can handle it.