IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2302.03172.html
   My bibliography  Save this paper

High-Dimensional Conditionally Gaussian State Space Models with Missing Data

Author

Listed:
  • Joshua C. C. Chan
  • Aubrey Poon
  • Dan Zhu

Abstract

We develop an efficient sampling approach for handling complex missing data patterns and a large number of missing observations in conditionally Gaussian state space models. Two important examples are dynamic factor models with unbalanced datasets and large Bayesian VARs with variables in multiple frequencies. A key insight underlying the proposed approach is that the joint distribution of the missing data conditional on the observed data is Gaussian. Moreover, the inverse covariance or precision matrix of this conditional distribution is sparse, and this special structure can be exploited to substantially speed up computations. We illustrate the methodology using two empirical applications. The first application combines quarterly, monthly and weekly data using a large Bayesian VAR to produce weekly GDP estimates. In the second application, we extract latent factors from unbalanced datasets involving over a hundred monthly variables via a dynamic factor model with stochastic volatility.

Suggested Citation

  • Joshua C. C. Chan & Aubrey Poon & Dan Zhu, 2023. "High-Dimensional Conditionally Gaussian State Space Models with Missing Data," Papers 2302.03172, arXiv.org.
  • Handle: RePEc:arx:papers:2302.03172
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2302.03172
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Chan, Joshua C.C. & Eisenstat, Eric & Koop, Gary, 2016. "Large Bayesian VARMAs," Journal of Econometrics, Elsevier, vol. 192(2), pages 374-390.
    2. James Mitchell & Gary Koop & Stuart McIntyre & Aubrey Poon, 2020. "Reconciled Estimates of Monthly GDP in the US," Economic Statistics Centre of Excellence (ESCoE) Discussion Papers ESCoE DP-2020-16, Economic Statistics Centre of Excellence (ESCoE).
    3. Angelia L. Grant & Joshua C.C. Chan, 2017. "A Bayesian Model Comparison for Trend‐Cycle Decompositions of Output," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 49(2-3), pages 525-552, March.
    4. Chan, Joshua C.C., 2023. "Comparing stochastic volatility specifications for large Bayesian VARs," Journal of Econometrics, Elsevier, vol. 235(2), pages 1419-1446.
    5. Jushan Bai & Serena Ng, 2002. "Determining the Number of Factors in Approximate Factor Models," Econometrica, Econometric Society, vol. 70(1), pages 191-221, January.
    6. Kastner, Gregor, 2019. "Sparse Bayesian time-varying covariance estimation in many dimensions," Journal of Econometrics, Elsevier, vol. 210(1), pages 98-115.
    7. Michael W. McCracken & Serena Ng, 2016. "FRED-MD: A Monthly Database for Macroeconomic Research," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(4), pages 574-589, October.
    8. Michele Lenza & Giorgio E. Primiceri, 2022. "How to estimate a vector autoregression after March 2020," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(4), pages 688-699, June.
    9. Grant, Angelia L. & Chan, Joshua C.C., 2017. "Reconciling output gaps: Unobserved components model and Hodrick–Prescott filter," Journal of Economic Dynamics and Control, Elsevier, vol. 75(C), pages 114-121.
    10. Jarociński, Marek, 2015. "A note on implementing the Durbin and Koopman simulation smoother," Computational Statistics & Data Analysis, Elsevier, vol. 91(C), pages 1-3.
    11. James H. Stock & Mark W. Watson, 2016. "Core Inflation and Trend Inflation," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 770-784, October.
    12. McCausland, William J. & Miller, Shirley & Pelletier, Denis, 2011. "Simulation smoothing for state-space models: A computational efficiency analysis," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 199-212, January.
    13. Zhang, Bo & Chan, Joshua C.C. & Cross, Jamie L., 2020. "Stochastic volatility models with ARMA innovations: An application to G7 inflation forecasts," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1318-1328.
    14. Jushan Bai & Serena Ng, 2021. "Matrix Completion, Counterfactuals, and Factor Analysis of Missing Data," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1746-1763, October.
    15. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino & Elmar Mertens, 2024. "Addressing COVID-19 Outliers in BVARs with Stochastic Volatility," The Review of Economics and Statistics, MIT Press, vol. 106(5), pages 1403-1417, September.
    16. Timothy Cogley & Thomas J. Sargent, 2005. "Drift and Volatilities: Monetary Policies and Outcomes in the Post WWII U.S," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 8(2), pages 262-302, April.
    17. Antonello D'Agostino & Luca Gambetti & Domenico Giannone, 2013. "Macroeconomic forecasting and structural change," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(1), pages 82-101, January.
    18. Gary M. Koop, 2013. "Forecasting with Medium and Large Bayesian VARS," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(2), pages 177-203, March.
    19. Stefanos Dimitrakopoulos & Michalis Kolossiatis, 2020. "Bayesian analysis of moving average stochastic volatility models: modeling in-mean effects and leverage for financial time series," Econometric Reviews, Taylor & Francis Journals, vol. 39(4), pages 319-343, April.
    20. Joshua C. C. Chan & Gary Koop & Simon M. Potter, 2013. "A New Model of Trend Inflation," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 31(1), pages 94-106, January.
    21. repec:ulb:ulbeco:2013/13388 is not listed on IDEAS
    22. Joshua C. C. Chan, 2017. "The Stochastic Volatility in Mean Model With Time-Varying Parameters: An Application to Inflation Modeling," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 35(1), pages 17-28, January.
    23. Aruoba, S. BoraÄŸan & Diebold, Francis X. & Scotti, Chiara, 2009. "Real-Time Measurement of Business Conditions," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 417-427.
    24. Marta Banbura & Domenico Giannone & Lucrezia Reichlin, 2010. "Large Bayesian vector auto regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(1), pages 71-92.
    25. Litterman, Robert B, 1986. "Forecasting with Bayesian Vector Autoregressions-Five Years of Experience," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 25-38, January.
    26. Cross, Jamie & Poon, Aubrey, 2016. "Forecasting structural change and fat-tailed events in Australian macroeconomic variables," Economic Modelling, Elsevier, vol. 58(C), pages 34-51.
    27. Todd E. Clark, 2011. "Real-Time Density Forecasts From Bayesian Vector Autoregressions With Stochastic Volatility," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 327-341, July.
    28. Kaufmann, Sylvia & Schumacher, Christian, 2019. "Bayesian estimation of sparse dynamic factor models with order-independent and ex-post mode identification," Journal of Econometrics, Elsevier, vol. 210(1), pages 116-134.
    29. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino, 2016. "Common Drifting Volatility in Large Bayesian VARs," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(3), pages 375-390, July.
    30. Gary Koop & Stuart McIntyre & James Mitchell & Aubrey Poon, 2020. "Regional output growth in the United Kingdom: More timely and higher frequency estimates from 1970," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(2), pages 176-197, March.
    31. Chan, Joshua C.C., 2013. "Moving average stochastic volatility models with application to inflation forecast," Journal of Econometrics, Elsevier, vol. 176(2), pages 162-172.
    32. Giorgio E. Primiceri, 2005. "Time Varying Structural Vector Autoregressions and Monetary Policy," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(3), pages 821-852.
    33. Frank Schorfheide & Dongho Song, 2015. "Real-Time Forecasting With a Mixed-Frequency VAR," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(3), pages 366-380, July.
    34. Joshua C. C. Chan, 2020. "Large Bayesian VARs: A Flexible Kronecker Error Covariance Structure," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 68-79, January.
    35. Lenza, Michele & Primiceri, Giorgio E., 2020. "How to estimate a VAR after March 2020," Working Paper Series 2461, European Central Bank.
    36. Roberto S. Mariano & Yasutomo Murasawa, 2003. "A new coincident index of business cycles based on monthly and quarterly series," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 18(4), pages 427-443.
    37. Brave, Scott A. & Butters, R. Andrew & Justiniano, Alejandro, 2019. "Forecasting economic activity with mixed frequency BVARs," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1692-1707.
    38. Marta Banbura & Domenico Giannone & Lucrezia Reichlin, 2010. "Large Bayesian vector auto regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(1), pages 71-92.
    39. Florian Eckert & Philipp Kronenberg & Heiner Mikosch & Stefan Neuwirth, 2020. "Tracking Economic Activity With Alternative High-Frequency Data," KOF Working papers 20-488, KOF Swiss Economic Institute, ETH Zurich.
    40. Pettenuzzo, Davide & Sabbatucci, Riccardo & Timmermann, Allan, 2023. "Dividend suspensions and cash flows during the Covid-19 pandemic: A dynamic econometric model," Journal of Econometrics, Elsevier, vol. 235(2), pages 1522-1541.
    41. Hauber, Philipp & Schumacher, Christian, 2021. "Precision-based sampling with missing observations: A factor model application," Discussion Papers 11/2021, Deutsche Bundesbank.
    42. Daniel J. Lewis & Karel Mertens & James H. Stock & Mihir Trivedi, 2022. "Measuring real activity using a weekly economic index," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(4), pages 667-687, June.
    43. Petrella, Ivan & Antolin-Diaz, Juan & Drechsel, Thomas, 2021. "Advances in Nowcasting Economic Activity: Secular Trends, Large Shocks and New Data," CEPR Discussion Papers 15926, C.E.P.R. Discussion Papers.
    44. Roberto S. Mariano & Yasutomo Murasawa, 2010. "A Coincident Index, Common Factors, and Monthly Real GDP," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 72(1), pages 27-46, February.
    45. Gregor Kastner & Florian Huber, 2020. "Sparse Bayesian vector autoregressions in huge dimensions," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 39(7), pages 1142-1165, November.
    46. Carriero, Andrea & Clark, Todd E. & Marcellino, Massimiliano, 2019. "Large Bayesian vector autoregressions with stochastic volatility and non-conjugate priors," Journal of Econometrics, Elsevier, vol. 212(1), pages 137-154.
    47. J. Durbin, 2002. "A simple and efficient simulation smoother for state space time series analysis," Biometrika, Biometrika Trust, vol. 89(3), pages 603-616, August.
    48. Serena Ng & Susannah Scanlan, 2023. "Constructing High Frequency Economic Indicators by Imputation," Papers 2303.01863, arXiv.org, revised Oct 2023.
    49. Chib, Siddhartha & Jeliazkov, Ivan, 2006. "Inference in Semiparametric Dynamic Models for Binary Longitudinal Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 685-700, June.
    50. Fu, Bowen, 2020. "Is the slope of the Phillips curve time-varying? Evidence from unobserved components models," Economic Modelling, Elsevier, vol. 88(C), pages 320-340.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Eraslan, Sercan & Reif, Magnus, 2023. "A latent weekly GDP indicator for Germany," Technical Papers 08/2023, Deutsche Bundesbank.
    2. Mertens, Elmar, 2023. "Precision-based sampling for state space models that have no measurement error," Journal of Economic Dynamics and Control, Elsevier, vol. 154(C).
    3. Antolín-Díaz, Juan & Drechsel, Thomas & Petrella, Ivan, 2024. "Advances in nowcasting economic activity: The role of heterogeneous dynamics and fat tails," Journal of Econometrics, Elsevier, vol. 238(2).
    4. Iacopini, Matteo & Poon, Aubrey & Rossini, Luca & Zhu, Dan, 2023. "Bayesian mixed-frequency quantile vector autoregression: Eliciting tail risks of monthly US GDP," Journal of Economic Dynamics and Control, Elsevier, vol. 157(C).
    5. Joshua Chan, 2023. "BVARs and Stochastic Volatility," Papers 2310.14438, arXiv.org.
    6. Luca Barbaglia & Lorenzo Frattarolo & Niko Hauzenberger & Dominik Hirschbuehl & Florian Huber & Luca Onorante & Michael Pfarrhofer & Luca Tiozzo Pezzoli, 2024. "Nowcasting economic activity in European regions using a mixed-frequency dynamic factor model," Papers 2401.10054, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chan, Joshua C.C., 2023. "Comparing stochastic volatility specifications for large Bayesian VARs," Journal of Econometrics, Elsevier, vol. 235(2), pages 1419-1446.
    2. Joshua Chan, 2023. "BVARs and Stochastic Volatility," Papers 2310.14438, arXiv.org.
    3. Joshua C.C. Chan & Rodney W. Strachan, 2023. "Bayesian State Space Models In Macroeconometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 37(1), pages 58-75, February.
    4. Sebastian Ankargren & Paulina Jon'eus, 2019. "Estimating Large Mixed-Frequency Bayesian VAR Models," Papers 1912.02231, arXiv.org.
    5. Gael M. Martin & David T. Frazier & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2023. "Bayesian Forecasting in the 21st Century: A Modern Review," Monash Econometrics and Business Statistics Working Papers 1/23, Monash University, Department of Econometrics and Business Statistics.
    6. Joshua C. C. Chan, 2019. "Large Bayesian vector autoregressions," CAMA Working Papers 2019-19, Centre for Applied Macroeconomic Analysis, Crawford School of Public Policy, The Australian National University.
    7. Martin, Gael M. & Frazier, David T. & Maneesoonthorn, Worapree & Loaiza-Maya, Rubén & Huber, Florian & Koop, Gary & Maheu, John & Nibbering, Didier & Panagiotelis, Anastasios, 2024. "Bayesian forecasting in economics and finance: A modern review," International Journal of Forecasting, Elsevier, vol. 40(2), pages 811-839.
    8. Ankargren, Sebastian & Jonéus, Paulina, 2021. "Simulation smoothing for nowcasting with large mixed-frequency VARs," Econometrics and Statistics, Elsevier, vol. 19(C), pages 97-113.
    9. Chan, Joshua C.C., 2021. "Minnesota-type adaptive hierarchical priors for large Bayesian VARs," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1212-1226.
    10. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino & Elmar Mertens, 2024. "Addressing COVID-19 Outliers in BVARs with Stochastic Volatility," The Review of Economics and Statistics, MIT Press, vol. 106(5), pages 1403-1417, September.
    11. Chan, Joshua C.C. & Yu, Xuewen, 2022. "Fast and Accurate Variational Inference for Large Bayesian VARs with Stochastic Volatility," Journal of Economic Dynamics and Control, Elsevier, vol. 143(C).
    12. Zhang, Bo & Nguyen, Bao H., 2020. "Real-time forecasting of the Australian macroeconomy using Bayesian VARs," Working Papers 2020-12, University of Tasmania, Tasmanian School of Business and Economics.
    13. Magnus Reif, 2020. "Macroeconomics, Nonlinearities, and the Business Cycle," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 87.
    14. James Mitchell & Gary Koop & Stuart McIntyre & Aubrey Poon, 2020. "Reconciled Estimates of Monthly GDP in the US," Economic Statistics Centre of Excellence (ESCoE) Discussion Papers ESCoE DP-2020-16, Economic Statistics Centre of Excellence (ESCoE).
    15. Mertens, Elmar, 2023. "Precision-based sampling for state space models that have no measurement error," Journal of Economic Dynamics and Control, Elsevier, vol. 154(C).
    16. Joshua C. C. Chan & Eric Eisenstat & Chenghan Hou & Gary Koop, 2020. "Composite likelihood methods for large Bayesian VARs with stochastic volatility," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(6), pages 692-711, September.
    17. Chenghan Hou & Bao Nguyen & Bo Zhang, 2023. "Real‐time forecasting of the Australian macroeconomy using flexible Bayesian VARs," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 418-451, March.
    18. Tsionas, Mike G. & Izzeldin, Marwan & Trapani, Lorenzo, 2022. "Estimation of large dimensional time varying VARs using copulas," European Economic Review, Elsevier, vol. 141(C).
    19. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino, 2015. "Realtime nowcasting with a Bayesian mixed frequency model with stochastic volatility," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 837-862, October.
    20. Hauzenberger, Niko, 2021. "Flexible Mixture Priors for Large Time-varying Parameter Models," Econometrics and Statistics, Elsevier, vol. 20(C), pages 87-108.

    More about this item

    JEL classification:

    • C11 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Bayesian Analysis: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2302.03172. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.