IDEAS home Printed from https://ideas.repec.org/p/msh/ebswps/2022-14.html
   My bibliography  Save this paper

Computing Bayes: From Then `Til Now

Author

Listed:
  • Gael M. Martin
  • David T. Frazier
  • Christian P. Robert

Abstract

This paper takes the reader on a journey through the history of Bayesian computation, from the 18th century to the present day. Beginning with the one-dimensional integral first confronted by Bayes in 1763, we highlight the key contributions of: Laplace, Metropolis (and, importantly, his coauthors!), Hammersley and Handscomb, and Hastings, all of which set the foundations for the computational revolution in the late 20th century -- led, primarily, by Markov chain Monte Carlo (MCMC) algorithms. A very short outline of 21st century computational methods -- including pseudo-marginal MCMC, Hamiltonian Monte Carlo, sequential Monte Carlo, and the various `approximate' methods -- completes the paper.

Suggested Citation

  • Gael M. Martin & David T. Frazier & Christian P. Robert, 2022. "Computing Bayes: From Then `Til Now," Monash Econometrics and Business Statistics Working Papers 14/22, Monash University, Department of Econometrics and Business Statistics.
    • Handle: RePEc:msh:ebswps:2022-14
    as

    Download full text from publisher

    File URL: https://www.monash.edu/business/ebs/research/publications/ebs/wp14-2022.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Luke Bornn & Neil Shephard & Reza Solgi, 2019. "Moment conditions and Bayesian non‐parametrics," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 81(1), pages 5-43, February.
    2. Geweke, John, 1989. "Bayesian Inference in Econometric Models Using Monte Carlo Integration," Econometrica, Econometric Society, vol. 57(6), pages 1317-1339, November.
    3. David M. Blei & Alp Kucukelbir & Jon D. McAuliffe, 2017. "Variational Inference: A Review for Statisticians," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(518), pages 859-877, April.
    4. David T. Frazier & Ruben Loaiza-Maya & Gael M. Martin & Bonsoo Koo, 2021. "Loss-Based Variational Bayes Prediction," Papers 2104.14054, arXiv.org, revised May 2022.
    5. Carpenter, Bob & Gelman, Andrew & Hoffman, Matthew D. & Lee, Daniel & Goodrich, Ben & Betancourt, Michael & Brubaker, Marcus & Guo, Jiqiang & Li, Peter & Riddell, Allen, 2017. "Stan: A Probabilistic Programming Language," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i01).
    6. David J. Nott & Robert Kohn, 2005. "Adaptive sampling for Bayesian variable selection," Biometrika, Biometrika Trust, vol. 92(4), pages 747-763, December.
    7. Matias Quiroz & Robert Kohn & Mattias Villani & Minh-Ngoc Tran, 2019. "Speeding Up MCMC by Efficient Data Subsampling," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(526), pages 831-843, April.
    8. Chib, Siddhartha & Greenberg, Edward, 1996. "Markov Chain Monte Carlo Simulation Methods in Econometrics," Econometric Theory, Cambridge University Press, vol. 12(3), pages 409-431, August.
    9. Jacquier, Eric & Polson, Nicholas G & Rossi, Peter E, 2002. "Bayesian Analysis of Stochastic Volatility Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 69-87, January.
    10. Geweke, John & Koop, Gary & van Dijk, Herman (ed.), 2011. "The Oxford Handbook of Bayesian Econometrics," OUP Catalogue, Oxford University Press, number 9780199559084, Decembrie.
    11. Kloek, Tuen & van Dijk, Herman K, 1978. "Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte Carlo," Econometrica, Econometric Society, vol. 46(1), pages 1-19, January.
    12. Gareth O. Roberts & Jeffrey S. Rosenthal, 1999. "Convergence of Slice Sampler Markov Chains," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 61(3), pages 643-660.
    13. repec:dau:papers:123456789/6072 is not listed on IDEAS
    14. Laurent E. Calvet & Veronika Czellar, 2015. "Accurate Methods for Approximate Bayesian Computation Filtering," Journal of Financial Econometrics, Oxford University Press, vol. 13(4), pages 798-838.
    15. Håvard Rue & Sara Martino & Nicolas Chopin, 2009. "Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 71(2), pages 319-392, April.
    16. Sangjoon Kim & Neil Shephard & Siddhartha Chib, 1998. "Stochastic Volatility: Likelihood Inference and Comparison with ARCH Models," Review of Economic Studies, Oxford University Press, vol. 65(3), pages 361-393.
    17. Ardia, David & Baştürk, Nalan & Hoogerheide, Lennart & van Dijk, Herman K., 2012. "A comparative study of Monte Carlo methods for efficient evaluation of marginal likelihood," Computational Statistics & Data Analysis, Elsevier, vol. 56(11), pages 3398-3414.
    18. repec:dau:papers:123456789/3578 is not listed on IDEAS
    19. Thomas A. Dean & Sumeetpal S. Singh & Ajay Jasra & Gareth W. Peters, 2014. "Parameter Estimation for Hidden Markov Models with Intractable Likelihoods," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 970-987, December.
    20. Petros Dellaportas & Ioannis Kontoyiannis, 2012. "Control variates for estimation based on reversible Markov chain Monte Carlo samplers," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 74(1), pages 133-161, January.
    21. Xin Luo & Håkon Tjelmeland, 2019. "A multiple-try Metropolis–Hastings algorithm with tailored proposals," Computational Statistics, Springer, vol. 34(3), pages 1109-1133, September.
    22. Chib, Siddhartha & Ramamurthy, Srikanth, 2010. "Tailored randomized block MCMC methods with application to DSGE models," Journal of Econometrics, Elsevier, vol. 155(1), pages 19-38, March.
    23. Bierkens, Joris & Bouchard-Côté, Alexandre & Doucet, Arnaud & Duncan, Andrew B. & Fearnhead, Paul & Lienart, Thibaut & Roberts, Gareth & Vollmer, Sebastian J., 2018. "Piecewise deterministic Markov processes for scalable Monte Carlo on restricted domains," Statistics & Probability Letters, Elsevier, vol. 136(C), pages 148-154.
    24. Ajay Jasra, 2015. "Approximate Bayesian Computation for a Class of Time Series Models," International Statistical Review, International Statistical Institute, vol. 83(3), pages 405-435, December.
    25. repec:dau:papers:123456789/3549 is not listed on IDEAS
    26. P. Damlen & J. Wakefield & S. Walker, 1999. "Gibbs sampling for Bayesian non‐conjugate and hierarchical models by using auxiliary variables," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 61(2), pages 331-344, April.
    27. David T. Frazier & Christian P. Robert & Judith Rousseau, 2020. "Model misspecification in approximate Bayesian computation: consequences and diagnostics," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(2), pages 421-444, April.
    28. Jacquier, Eric & Polson, Nicholas G & Rossi, Peter E, 1994. "Bayesian Analysis of Stochastic Volatility Models: Comments: Reply," Journal of Business & Economic Statistics, American Statistical Association, vol. 12(4), pages 413-417, October.
    29. repec:dau:papers:123456789/6189 is not listed on IDEAS
    30. Murray Pollock & Paul Fearnhead & Adam M. Johansen & Gareth O. Roberts, 2020. "Quasi‐stationary Monte Carlo and the ScaLE algorithm," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(5), pages 1167-1221, December.
    31. Gael M. Martin & David T. Frazier & Christian P. Robert, 2020. "Computing Bayes: Bayesian Computation from 1763 to the 21st Century," Monash Econometrics and Business Statistics Working Papers 14/20, Monash University, Department of Econometrics and Business Statistics.
    32. Ormerod, J. T. & Wand, M. P., 2010. "Explaining Variational Approximations," The American Statistician, American Statistical Association, vol. 64(2), pages 140-153.
    33. Gael M. Martin & David T. Frazier & Christian P. Robert, 2021. "Approximating Bayes in the 21st Century," Monash Econometrics and Business Statistics Working Papers 24/21, Monash University, Department of Econometrics and Business Statistics.
    34. Gareth O. Roberts & Jeffrey S. Rosenthal, 1998. "Optimal scaling of discrete approximations to Langevin diffusions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 60(1), pages 255-268.
    35. Pitt, Michael K. & Silva, Ralph dos Santos & Giordani, Paolo & Kohn, Robert, 2012. "On some properties of Markov chain Monte Carlo simulation methods based on the particle filter," Journal of Econometrics, Elsevier, vol. 171(2), pages 134-151.
    36. Mathieu Gerber & Nicolas Chopin, 2015. "Sequential quasi Monte Carlo," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 77(3), pages 509-579, June.
    37. G. O. Roberts & S. K. Sahu, 1997. "Updating Schemes, Correlation Structure, Blocking and Parameterization for the Gibbs Sampler," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 59(2), pages 291-317.
    38. Giacomo Zanella & Gareth Roberts, 2019. "Scalable importance tempering and Bayesian variable selection," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 81(3), pages 489-517, July.
    39. George Deligiannidis & Arnaud Doucet & Michael K. Pitt, 2018. "The correlated pseudomarginal method," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(5), pages 839-870, November.
    40. Pierre E. Jacob & John O’Leary & Yves F. Atchadé, 2020. "Unbiased Markov chain Monte Carlo methods with couplings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 543-600, July.
    41. A. Doucet & M. K. Pitt & G. Deligiannidis & R. Kohn, 2015. "Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator," Biometrika, Biometrika Trust, vol. 102(2), pages 295-313.
    42. Mark A. Beaumont & Jean-Marie Cornuet & Jean-Michel Marin & Christian P. Robert, 2009. "Adaptive approximate Bayesian computation," Biometrika, Biometrika Trust, vol. 96(4), pages 983-990.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gael M. Martin & David T. Frazier & Worapree Maneesoonthorn & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2022. "Bayesian Forecasting in Economics and Finance: A Modern Review," Papers 2212.03471, arXiv.org, revised Jul 2023.
    2. Gael M. Martin & David T. Frazier & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2023. "Bayesian Forecasting in the 21st Century: A Modern Review," Monash Econometrics and Business Statistics Working Papers 1/23, Monash University, Department of Econometrics and Business Statistics.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gael M. Martin & David T. Frazier & Christian P. Robert, 2020. "Computing Bayes: Bayesian Computation from 1763 to the 21st Century," Monash Econometrics and Business Statistics Working Papers 14/20, Monash University, Department of Econometrics and Business Statistics.
    2. Gael M. Martin & David T. Frazier & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2023. "Bayesian Forecasting in the 21st Century: A Modern Review," Monash Econometrics and Business Statistics Working Papers 1/23, Monash University, Department of Econometrics and Business Statistics.
    3. Gael M. Martin & David T. Frazier & Christian P. Robert, 2021. "Approximating Bayes in the 21st Century," Monash Econometrics and Business Statistics Working Papers 24/21, Monash University, Department of Econometrics and Business Statistics.
    4. Gael M. Martin & David T. Frazier & Worapree Maneesoonthorn & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2022. "Bayesian Forecasting in Economics and Finance: A Modern Review," Papers 2212.03471, arXiv.org, revised Jul 2023.
    5. Nalan Basturk & Cem Cakmakli & S. Pinar Ceyhan & Herman K. van Dijk, 2014. "On the Rise of Bayesian Econometrics after Cowles Foundation Monographs 10, 14," Tinbergen Institute Discussion Papers 14-085/III, Tinbergen Institute, revised 04 Sep 2014.
    6. David T. Frazier & Ruben Loaiza-Maya & Gael M. Martin, 2021. "Variational Bayes in State Space Models: Inferential and Predictive Accuracy," Papers 2106.12262, arXiv.org, revised Feb 2022.
    7. Dang, Khue-Dung & Quiroz, Matias & Kohn, Robert & Tran, Minh-Ngoc & Villani, Mattias, 2019. "Hamiltonian Monte Carlo with Energy Conserving Subsampling," Working Paper Series 372, Sveriges Riksbank (Central Bank of Sweden).
    8. Nalan Basturk & Cem Cakmakli & S. Pinar Ceyhan & Herman K. van Dijk, 2013. "Historical Developments in Bayesian Econometrics after Cowles Foundation Monographs 10, 14," Tinbergen Institute Discussion Papers 13-191/III, Tinbergen Institute.
    9. Chaya Weerasinghe & Ruben Loaiza-Maya & Gael M. Martin & David T. Frazier, 2023. "ABC-based Forecasting in State Space Models," Monash Econometrics and Business Statistics Working Papers 12/23, Monash University, Department of Econometrics and Business Statistics.
    10. Strickland, Chris M. & Martin, Gael M. & Forbes, Catherine S., 2008. "Parameterisation and efficient MCMC estimation of non-Gaussian state space models," Computational Statistics & Data Analysis, Elsevier, vol. 52(6), pages 2911-2930, February.
    11. Antonio A. F. Santos, 2021. "Bayesian Estimation for High-Frequency Volatility Models in a Time Deformed Framework," Computational Economics, Springer;Society for Computational Economics, vol. 57(2), pages 455-479, February.
    12. Matias Quiroz & Mattias Villani & Robert Kohn & Minh-Ngoc Tran & Khue-Dung Dang, 2018. "Subsampling MCMC - an Introduction for the Survey Statistician," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 80(1), pages 33-69, December.
    13. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    14. Dellaportas, Petros & Titsias, Michalis K. & Petrova, Katerina & Plataniotis, Anastasios, 2023. "Scalable inference for a full multivariate stochastic volatility model," Journal of Econometrics, Elsevier, vol. 232(2), pages 501-520.
    15. Siem Jan Koopman & Charles S. Bos, 2002. "Time Series Models with a Common Stochastic Variance for Analysing Economic Time Series," Tinbergen Institute Discussion Papers 02-113/4, Tinbergen Institute.
    16. Chib, Siddhartha, 2004. "Markov Chain Monte Carlo Technology," Papers 2004,22, Humboldt University of Berlin, Center for Applied Statistics and Economics (CASE).
    17. Matti Vihola & Jouni Helske & Jordan Franks, 2020. "Importance sampling type estimators based on approximate marginal Markov chain Monte Carlo," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(4), pages 1339-1376, December.
    18. McCausland, William J., 2012. "The HESSIAN method: Highly efficient simulation smoothing, in a nutshell," Journal of Econometrics, Elsevier, vol. 168(2), pages 189-206.
    19. Hajargasht, Gholamreza & Rao, D.S. Prasada, 2019. "Multilateral index number systems for international price comparisons: Properties, existence and uniqueness," Journal of Mathematical Economics, Elsevier, vol. 83(C), pages 36-47.
    20. Liesenfeld, Roman & Richard, Jean-François, 2008. "Improving MCMC, using efficient importance sampling," Computational Statistics & Data Analysis, Elsevier, vol. 53(2), pages 272-288, December.

    More about this item

    Keywords

    History of Bayesian computation; Laplace approximation; Metropolis-Hastings algorithm; importance sampling; Markov chain Monte Carlo; pseudo-marginal methods; Hamiltonian Monte Carlo; sequential Monte Carlo; approximate Bayesian methods;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:msh:ebswps:2022-14. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Professor Xibin Zhang (email available below). General contact details of provider: https://edirc.repec.org/data/dxmonau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.