IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_5290.html
   My bibliography  Save this paper

Asymptotic Variance of Brier (Skill) Score in the Presence of Serial Correlation

Author

Listed:
  • Kajal Lahiri
  • Liu Yang

Abstract

We derive autocorrelation-robust asymptotic variances of the Brier score and Brier skill score, which are generally applicable in circumstances with weak serial correlation. A simulation experiment and an empirical application from macroeconomics underscore the importance of taking care of serial correlation. We find that the conventional variances are too conservative to account for the sampling variability in estimating the Brier (skill) score.

Suggested Citation

  • Kajal Lahiri & Liu Yang, 2015. "Asymptotic Variance of Brier (Skill) Score in the Presence of Serial Correlation," CESifo Working Paper Series 5290, CESifo.
  • Handle: RePEc:ces:ceswps:_5290
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp5290.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Gad Levanon & Jean-Claude Manini & Ataman Ozyildirim & Brian Schaitkin & Jennelyn Tanchua, 2011. "Using a Leading Credit Index to Predict Turning Points in the U.S. Business Cycle," Economics Program Working Papers 11-05, The Conference Board, Economics Program.
    2. Lahiri, Kajal & Monokroussos, George & Zhao, Yongchen, 2013. "The yield spread puzzle and the information content of SPF forecasts," Economics Letters, Elsevier, vol. 118(1), pages 219-221.
    3. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," Review of Economic Studies, Oxford University Press, vol. 61(4), pages 631-653.
    4. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    5. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    6. Lahiri, Kajal & Yang, Liu, 2016. "Asymptotic variance of Brier (skill) score in the presence of serial correlation," Economics Letters, Elsevier, vol. 141(C), pages 125-129.
    7. Lopez, Jose A, 2001. "Evaluating the Predictive Accuracy of Volatility Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 20(2), pages 87-109, March.
    8. Andrews, Donald W K & Monahan, J Christopher, 1992. "An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator," Econometrica, Econometric Society, vol. 60(4), pages 953-966, July.
    9. Pesaran, M. Hashem & Timmermann, Allan, 2009. "Testing Dependence Among Serially Correlated Multicategory Variables," Journal of the American Statistical Association, American Statistical Association, vol. 104(485), pages 325-337.
    10. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    11. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    12. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    13. Levanon, Gad & Manini, Jean-Claude & Ozyildirim, Ataman & Schaitkin, Brian & Tanchua, Jennelyn, 2015. "Using financial indicators to predict turning points in the business cycle: The case of the leading economic index for the United States," International Journal of Forecasting, Elsevier, vol. 31(2), pages 426-445.
    14. Rudebusch, Glenn D. & Williams, John C., 2009. "Forecasting Recessions: The Puzzle of the Enduring Power of the Yield Curve," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 492-503.
    15. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lahiri, Kajal & Yang, Liu, 2016. "Asymptotic variance of Brier (skill) score in the presence of serial correlation," Economics Letters, Elsevier, vol. 141(C), pages 125-129.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Liu, Weiling & Moench, Emanuel, 2016. "What predicts US recessions?," International Journal of Forecasting, Elsevier, vol. 32(4), pages 1138-1150.
    2. Kajal Lahiri & Liu Yang, 2018. "Confidence Bands for ROC Curves With Serially Dependent Data," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(1), pages 115-130, January.
    3. Lahiri, Kajal & Peng, Huaming & Zhao, Yongchen, 2015. "Testing the value of probability forecasts for calibrated combining," International Journal of Forecasting, Elsevier, vol. 31(1), pages 113-129.
    4. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    5. Firmin Doko Tchatoka & Qazi Haque, 2020. "On bootstrapping tests of equal forecast accuracy for nested models," Economics Discussion / Working Papers 20-06, The University of Western Australia, Department of Economics.
    6. Moench, Emanuel & Stein, Tobias, 2021. "Equity premium predictability over the business cycle," CEPR Discussion Papers 16357, C.E.P.R. Discussion Papers.
    7. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    8. Henri Nyberg, 2018. "Forecasting US interest rates and business cycle with a nonlinear regime switching VAR model," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 37(1), pages 1-15, January.
    9. Hartigan, Luke, 2018. "Alternative HAC covariance matrix estimators with improved finite sample properties," Computational Statistics & Data Analysis, Elsevier, vol. 119(C), pages 55-73.
    10. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    11. Giacomini, Raffaella & Komunjer, Ivana, 2005. "Evaluation and Combination of Conditional Quantile Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 416-431, October.
    12. Blaskowitz, Oliver & Herwartz, Helmut, 2014. "Testing the value of directional forecasts in the presence of serial correlation," International Journal of Forecasting, Elsevier, vol. 30(1), pages 30-42.
    13. Ekaterini Panopoulou, 2005. "A Resolution of the Fisher Effect Puzzle: A Comparison of Estimators," Money Macro and Finance (MMF) Research Group Conference 2005 18, Money Macro and Finance Research Group.
    14. Wouter J. Den Haan & Andrew T. Levin, 1995. "Inferences from parametric and non-parametric covariance matrix estimation procedures," International Finance Discussion Papers 504, Board of Governors of the Federal Reserve System (U.S.).
    15. Marian Vavra, 2015. "On a Bootstrap Test for Forecast Evaluations," Working and Discussion Papers WP 5/2015, Research Department, National Bank of Slovakia.
    16. Ahmed, Shamim & Liu, Xiaoquan & Valente, Giorgio, 2016. "Can currency-based risk factors help forecast exchange rates?," International Journal of Forecasting, Elsevier, vol. 32(1), pages 75-97.
    17. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    18. Mertens, Elmar, 2012. "Are spectral estimators useful for long-run restrictions in SVARs?," Journal of Economic Dynamics and Control, Elsevier, vol. 36(12), pages 1831-1844.
    19. Michael Jansson & Marcelo J. Moreira, 2006. "Optimal Inference in Regression Models with Nearly Integrated Regressors," Econometrica, Econometric Society, vol. 74(3), pages 681-714, May.
    20. Härdle, Wolfgang & Horowitz, Joel L. & Kreiss, Jens-Peter, 2001. "Bootstrap methods for time series," SFB 373 Discussion Papers 2001,59, Humboldt University of Berlin, Interdisciplinary Research Project 373: Quantification and Simulation of Economic Processes.

    More about this item

    Keywords

    probability forecasts; serial correlation; Brier score; Brier skill score; survey of professional forecasters;
    All these keywords.

    JEL classification:

    • C01 - Mathematical and Quantitative Methods - - General - - - Econometrics
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_5290. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.