IDEAS home Printed from https://ideas.repec.org/p/nya/albaec/13-07.html
   My bibliography  Save this paper

Confidence Bands for ROC Curves with Serially Dependent Data

Author

Listed:
  • Kajal Lahiri
  • Liu Yang

Abstract

We propose serial correlation robust asymptotic confidence bands for the receiver operating characteristic (ROC) curves estimated by quasi-maximum likelihood in the binormal model. Our simulation experiments confirm that this new method performs fairly well in finite samples. The conventional procedure is found to be markedly undersized in terms of yielding empirical coverage probabilities lower than the nominal level, especially when the serial correlation is strong. We evaluate the three-quarter-ahead probability forecasts for real GDP declines from the Survey of Professional Forecasters, and find that one would draw a misleading conclusion about forecasting skill if serial correlation is ignored.

Suggested Citation

  • Kajal Lahiri & Liu Yang, 2013. "Confidence Bands for ROC Curves with Serially Dependent Data," Discussion Papers 13-07, University at Albany, SUNY, Department of Economics.
    • Handle: RePEc:nya:albaec:13-07
    as

    Download full text from publisher

    File URL: http://www.albany.edu/economics/research/workingp/2013/ly_roc.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Peter G. Hall & Rob J. Hyndman & Yanan Fan, 2003. "Non Parametric Confidence Intervals for Receiver Operating Characteristic Curves," Monash Econometrics and Business Statistics Working Papers 12/03, Monash University, Department of Econometrics and Business Statistics.
    2. Gorr, Wilpen L. & Schneider, Matthew J., 2013. "Large-change forecast accuracy: Reanalysis of M3-Competition data using receiver operating characteristic analysis," International Journal of Forecasting, Elsevier, vol. 29(2), pages 274-281.
    3. Yixiao Sun, 2013. "A heteroskedasticity and autocorrelation robust F test using an orthonormal series variance estimator," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-26, February.
    4. James A. Hanley, 1988. "The Robustness of the "Binormal" Assumptions Used in Fitting ROC Curves," Medical Decision Making, , vol. 8(3), pages 197-203, August.
    5. Yixiao Sun & Peter C. B. Phillips & Sainan Jin, 2008. "Optimal Bandwidth Selection in Heteroskedasticity-Autocorrelation Robust Testing," Econometrica, Econometric Society, vol. 76(1), pages 175-194, January.
    6. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    7. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    8. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    9. Òscar Jordà & Moritz Schularick & Alan M Taylor, 2011. "Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 59(2), pages 340-378, June.
    10. Sun, Yixiao, 2014. "Let’s fix it: Fixed-b asymptotics versus small-b asymptotics in heteroskedasticity and autocorrelation robust inference," Journal of Econometrics, Elsevier, vol. 178(P3), pages 659-677.
    11. Peter Hall, 2004. "Nonparametric confidence intervals for receiver operating characteristic curves," Biometrika, Biometrika Trust, vol. 91(3), pages 743-750, September.
    12. Stephen Satchel & Wei Xia, 2006. "Analytic Models of the ROC Curve: Applications to Credit Rating Model Validation," Research Paper Series 181, Quantitative Finance Research Centre, University of Technology, Sydney.
    13. King, Gary & Zeng, Langche, 2001. "Logistic Regression in Rare Events Data," Political Analysis, Cambridge University Press, vol. 9(2), pages 137-163, January.
    14. Cohen, Jacqueline & Garman, Samuel & Gorr, Wilpen, 2009. "Empirical calibration of time series monitoring methods using receiver operating characteristic curves," International Journal of Forecasting, Elsevier, vol. 25(3), pages 484-497, July.
    15. Travis J. Berge & Òscar Jordà, 2011. "Evaluating the Classification of Economic Activity into Recessions and Expansions," American Economic Journal: Macroeconomics, American Economic Association, vol. 3(2), pages 246-277, April.
    16. Pesaran, M. Hashem & Timmermann, Allan, 2009. "Testing Dependence Among Serially Correlated Multicategory Variables," Journal of the American Statistical Association, American Statistical Association, vol. 104(485), pages 325-337.
    17. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2005. "A New Asymptotic Theory For Heteroskedasticity-Autocorrelation Robust Tests," Econometric Theory, Cambridge University Press, vol. 21(6), pages 1130-1164, December.
    18. Eugene Demidenko, 2012. "Confidence intervals and bands for the binormal ROC curve revisited," Journal of Applied Statistics, Taylor & Francis Journals, vol. 39(1), pages 67-79, March.
    19. Zeileis, Achim, 2004. "Econometric Computing with HC and HAC Covariance Matrix Estimators," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 11(i10).
    20. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    21. Blaskowitz, Oliver & Herwartz, Helmut, 2014. "Testing the value of directional forecasts in the presence of serial correlation," International Journal of Forecasting, Elsevier, vol. 30(1), pages 30-42.
    22. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    23. Oliver Blaskowitz & Helmut Herwartz, 2008. "Testing directional forecast value in the presence of serial correlation," SFB 649 Discussion Papers SFB649DP2008-073, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    24. Zeileis, Achim, 2006. "Object-oriented Computation of Sandwich Estimators," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 16(i09).
    25. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    26. Stein, Roger M., 2005. "The relationship between default prediction and lending profits: Integrating ROC analysis and loan pricing," Journal of Banking & Finance, Elsevier, vol. 29(5), pages 1213-1236, May.
    27. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    28. Blochlinger, Andreas & Leippold, Markus, 2006. "Economic benefit of powerful credit scoring," Journal of Banking & Finance, Elsevier, vol. 30(3), pages 851-873, March.
    29. Guangqin Ma & W.J. Hall, 1993. "Confidence Bands for Receiver Operating Characteristic Curves," Medical Decision Making, , vol. 13(3), pages 191-197, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lahiri Kajal & Yang Liu, 2016. "A non-linear forecast combination procedure for binary outcomes," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 20(4), pages 421-440, September.
    2. Máximo Camacho & Gonzalo Palmieri, 2021. "Evaluating the OECD’s main economic indicators at anticipating recessions," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(1), pages 80-93, January.
    3. Yu-Chin Hsu & Robert P. Lieli, 2021. "Inference for ROC Curves Based on Estimated Predictive Indices," Papers 2112.01772, arXiv.org.
    4. Kajal Lahiri & Liu Yang, 2023. "Predicting binary outcomes based on the pair-copula construction," Empirical Economics, Springer, vol. 64(6), pages 3089-3119, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bespalova, Olga, 2018. "Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA," MPRA Paper 117706, University Library of Munich, Germany.
    2. Casini, Alessandro, 2023. "Theory of evolutionary spectra for heteroskedasticity and autocorrelation robust inference in possibly misspecified and nonstationary models," Journal of Econometrics, Elsevier, vol. 235(2), pages 372-392.
    3. Lahiri, Kajal & Yang, Liu, 2016. "Asymptotic variance of Brier (skill) score in the presence of serial correlation," Economics Letters, Elsevier, vol. 141(C), pages 125-129.
    4. Kim, Min Seong & Sun, Yixiao & Yang, Jingjing, 2017. "A fixed-bandwidth view of the pre-asymptotic inference for kernel smoothing with time series data," Journal of Econometrics, Elsevier, vol. 197(2), pages 298-322.
    5. Martínez-Iriarte, Julián & Sun, Yixiao & Wang, Xuexin, 2020. "Asymptotic F tests under possibly weak identification," Journal of Econometrics, Elsevier, vol. 218(1), pages 140-177.
    6. Eben Lazarus & Daniel J. Lewis & James H. Stock, 2021. "The Size‐Power Tradeoff in HAR Inference," Econometrica, Econometric Society, vol. 89(5), pages 2497-2516, September.
    7. Kim, Min Seong & Sun, Yixiao, 2013. "Heteroskedasticity and spatiotemporal dependence robust inference for linear panel models with fixed effects," Journal of Econometrics, Elsevier, vol. 177(1), pages 85-108.
    8. Rho, Seunghwa & Vogelsang, Timothy J., 2021. "Inference in time series models using smoothed-clustered standard errors," Journal of Econometrics, Elsevier, vol. 224(1), pages 113-133.
    9. Hwang, Jungbin & Sun, Yixiao, 2018. "Should we go one step further? An accurate comparison of one-step and two-step procedures in a generalized method of moments framework," Journal of Econometrics, Elsevier, vol. 207(2), pages 381-405.
    10. Lu, Ye & Park, Joon Y., 2019. "Estimation of longrun variance of continuous time stochastic process using discrete sample," Journal of Econometrics, Elsevier, vol. 210(2), pages 236-267.
    11. Hwang, Jungbin & Sun, Yixiao, 2017. "Asymptotic F and t tests in an efficient GMM setting," Journal of Econometrics, Elsevier, vol. 198(2), pages 277-295.
    12. Preinerstorfer, David, 2014. "Finite Sample Properties of Tests Based on Prewhitened Nonparametric Covariance Estimators," MPRA Paper 58333, University Library of Munich, Germany.
    13. Máximo Camacho & Gonzalo Palmieri, 2021. "Evaluating the OECD’s main economic indicators at anticipating recessions," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(1), pages 80-93, January.
    14. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    15. Kai Wenger & Christian Leschinski & Philipp Sibbertsen, 2019. "Change-in-mean tests in long-memory time series: a review of recent developments," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 103(2), pages 237-256, June.
    16. Federico Belotti & Alessandro Casini & Leopoldo Catania & Stefano Grassi & Pierre Perron, 2023. "Simultaneous bandwidths determination for DK-HAC estimators and long-run variance estimation in nonparametric settings," Econometric Reviews, Taylor & Francis Journals, vol. 42(3), pages 281-306, February.
    17. Hirukawa, Masayuki, 2023. "Robust Covariance Matrix Estimation in Time Series: A Review," Econometrics and Statistics, Elsevier, vol. 27(C), pages 36-61.
    18. Boudoukh, Jacob & Israel, Ronen & Richardson, Matthew, 2022. "Biases in long-horizon predictive regressions," Journal of Financial Economics, Elsevier, vol. 145(3), pages 937-969.
    19. Hwang, Jungbin & Sun, Yixiao, 2018. "SIMPLE, ROBUST, AND ACCURATE F AND t TESTS IN COINTEGRATED SYSTEMS," Econometric Theory, Cambridge University Press, vol. 34(5), pages 949-984, October.
    20. Liu, Weiling & Moench, Emanuel, 2016. "What predicts US recessions?," International Journal of Forecasting, Elsevier, vol. 32(4), pages 1138-1150.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nya:albaec:13-07. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Byoung Park (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.