IDEAS home Printed from https://ideas.repec.org/p/nya/albaec/13-07.html
   My bibliography  Save this paper

Confidence Bands for ROC Curves with Serially Dependent Data

Author

Listed:
  • Kajal Lahiri
  • Liu Yang

Abstract

We propose serial correlation robust asymptotic confidence bands for the receiver operating characteristic (ROC) curves estimated by quasi-maximum likelihood in the binormal model. Our simulation experiments confirm that this new method performs fairly well in finite samples. The conventional procedure is found to be markedly undersized in terms of yielding empirical coverage probabilities lower than the nominal level, especially when the serial correlation is strong. We evaluate the three-quarter-ahead probability forecasts for real GDP declines from the Survey of Professional Forecasters, and find that one would draw a misleading conclusion about forecasting skill if serial correlation is ignored.

Suggested Citation

  • Kajal Lahiri & Liu Yang, 2013. "Confidence Bands for ROC Curves with Serially Dependent Data," Discussion Papers 13-07, University at Albany, SUNY, Department of Economics.
    • Handle: RePEc:nya:albaec:13-07
    as

    Download full text from publisher

    File URL: http://www.albany.edu/economics/research/workingp/2013/ly_roc.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Peter G. Hall & Rob J. Hyndman & Yanan Fan, 2003. "Non Parametric Confidence Intervals for Receiver Operating Characteristic Curves," Monash Econometrics and Business Statistics Working Papers 12/03, Monash University, Department of Econometrics and Business Statistics.
    2. Gorr, Wilpen L. & Schneider, Matthew J., 2013. "Large-change forecast accuracy: Reanalysis of M3-Competition data using receiver operating characteristic analysis," International Journal of Forecasting, Elsevier, vol. 29(2), pages 274-281.
    3. Yixiao Sun, 2013. "A heteroskedasticity and autocorrelation robust F test using an orthonormal series variance estimator," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-26, February.
    4. James A. Hanley, 1988. "The Robustness of the "Binormal" Assumptions Used in Fitting ROC Curves," Medical Decision Making, , vol. 8(3), pages 197-203, August.
    5. Yixiao Sun & Peter C. B. Phillips & Sainan Jin, 2008. "Optimal Bandwidth Selection in Heteroskedasticity-Autocorrelation Robust Testing," Econometrica, Econometric Society, vol. 76(1), pages 175-194, January.
    6. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    7. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    8. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    9. Òscar Jordà & Moritz Schularick & Alan M Taylor, 2011. "Financial Crises, Credit Booms, and External Imbalances: 140 Years of Lessons," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 59(2), pages 340-378, June.
    10. Sun, Yixiao, 2013. "Let's Fix It: Fixed-b Asymptotics versus Small-b Asymptotics in Heteroscedasticity and Autocorrelation Robust Inference," University of California at San Diego, Economics Working Paper Series qt8x8307rz, Department of Economics, UC San Diego.
    11. Peter Hall, 2004. "Nonparametric confidence intervals for receiver operating characteristic curves," Biometrika, Biometrika Trust, vol. 91(3), pages 743-750, September.
    12. Stephen Satchel & Wei Xia, 2006. "Analytic Models of the ROC Curve: Applications to Credit Rating Model Validation," Research Paper Series 181, Quantitative Finance Research Centre, University of Technology, Sydney.
    13. King, Gary & Zeng, Langche, 2001. "Logistic Regression in Rare Events Data," Political Analysis, Cambridge University Press, vol. 9(2), pages 137-163, January.
    14. Cohen, Jacqueline & Garman, Samuel & Gorr, Wilpen, 2009. "Empirical calibration of time series monitoring methods using receiver operating characteristic curves," International Journal of Forecasting, Elsevier, vol. 25(3), pages 484-497, July.
    15. Travis J. Berge & Òscar Jordà, 2011. "Evaluating the Classification of Economic Activity into Recessions and Expansions," American Economic Journal: Macroeconomics, American Economic Association, vol. 3(2), pages 246-277, April.
    16. Pesaran, M. Hashem & Timmermann, Allan, 2009. "Testing Dependence Among Serially Correlated Multicategory Variables," Journal of the American Statistical Association, American Statistical Association, vol. 104(485), pages 325-337.
    17. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2005. "A New Asymptotic Theory For Heteroskedasticity-Autocorrelation Robust Tests," Econometric Theory, Cambridge University Press, vol. 21(6), pages 1130-1164, December.
    18. Eugene Demidenko, 2012. "Confidence intervals and bands for the binormal ROC curve revisited," Journal of Applied Statistics, Taylor & Francis Journals, vol. 39(1), pages 67-79, March.
    19. Zeileis, Achim, 2004. "Econometric Computing with HC and HAC Covariance Matrix Estimators," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 11(i10).
    20. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    21. Blaskowitz, Oliver & Herwartz, Helmut, 2014. "Testing the value of directional forecasts in the presence of serial correlation," International Journal of Forecasting, Elsevier, vol. 30(1), pages 30-42.
    22. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    23. Oliver Blaskowitz & Helmut Herwartz, 2008. "Testing directional forecast value in the presence of serial correlation," SFB 649 Discussion Papers SFB649DP2008-073, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    24. Zeileis, Achim, 2006. "Object-oriented Computation of Sandwich Estimators," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 16(i09).
    25. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    26. Stein, Roger M., 2005. "The relationship between default prediction and lending profits: Integrating ROC analysis and loan pricing," Journal of Banking & Finance, Elsevier, vol. 29(5), pages 1213-1236, May.
    27. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    28. Blochlinger, Andreas & Leippold, Markus, 2006. "Economic benefit of powerful credit scoring," Journal of Banking & Finance, Elsevier, vol. 30(3), pages 851-873, March.
    29. Sun, Yixiao, 2014. "Let’s fix it: Fixed-b asymptotics versus small-b asymptotics in heteroskedasticity and autocorrelation robust inference," Journal of Econometrics, Elsevier, vol. 178(P3), pages 659-677.
    30. Guangqin Ma & W.J. Hall, 1993. "Confidence Bands for Receiver Operating Characteristic Curves," Medical Decision Making, , vol. 13(3), pages 191-197, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lahiri Kajal & Yang Liu, 2016. "A non-linear forecast combination procedure for binary outcomes," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 20(4), pages 421-440, September.
    2. Máximo Camacho & Gonzalo Palmieri, 2021. "Evaluating the OECD’s main economic indicators at anticipating recessions," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(1), pages 80-93, January.
    3. Yu-Chin Hsu & Robert P. Lieli, 2021. "Inference for ROC Curves Based on Estimated Predictive Indices," Papers 2112.01772, arXiv.org.
    4. Kajal Lahiri & Liu Yang, 2023. "Predicting binary outcomes based on the pair-copula construction," Empirical Economics, Springer, vol. 64(6), pages 3089-3119, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Casini, Alessandro, 2023. "Theory of evolutionary spectra for heteroskedasticity and autocorrelation robust inference in possibly misspecified and nonstationary models," Journal of Econometrics, Elsevier, vol. 235(2), pages 372-392.
    2. Bespalova, Olga, 2018. "Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA," MPRA Paper 117706, University Library of Munich, Germany.
    3. Kim, Min Seong & Sun, Yixiao & Yang, Jingjing, 2016. "A Fixed-bandwidth View of the Pre-asymptotic Inference for Kernel Smoothing with Time Series Data," University of California at San Diego, Economics Working Paper Series qt2240n3n5, Department of Economics, UC San Diego.
    4. Martínez-Iriarte, Julián & Sun, Yixiao & Wang, Xuexin, 2019. "Asymptotic F Tests under Possibly Weak Identification," University of California at San Diego, Economics Working Paper Series qt6qk200q8, Department of Economics, UC San Diego.
    5. Kim, Min Seong & Sun, Yixiao, 2013. "Heteroskedasticity and spatiotemporal dependence robust inference for linear panel models with fixed effects," Journal of Econometrics, Elsevier, vol. 177(1), pages 85-108.
    6. Rho, Seunghwa & Vogelsang, Timothy J., 2021. "Inference in time series models using smoothed-clustered standard errors," Journal of Econometrics, Elsevier, vol. 224(1), pages 113-133.
    7. Martínez-Iriarte, Julián & Sun, Yixiao & Wang, Xuexin, 2020. "Asymptotic F tests under possibly weak identification," Journal of Econometrics, Elsevier, vol. 218(1), pages 140-177.
    8. Hwang, Jungbin & Sun, Yixiao, 2015. "Should We Go One Step Further? An Accurate Comparison of One-step and Two-step Procedures in a Generalized Method of Moments Framework," University of California at San Diego, Economics Working Paper Series qt58r2z98m, Department of Economics, UC San Diego.
    9. Lahiri, Kajal & Yang, Liu, 2016. "Asymptotic variance of Brier (skill) score in the presence of serial correlation," Economics Letters, Elsevier, vol. 141(C), pages 125-129.
    10. Kim, Min Seong & Sun, Yixiao & Yang, Jingjing, 2017. "A fixed-bandwidth view of the pre-asymptotic inference for kernel smoothing with time series data," Journal of Econometrics, Elsevier, vol. 197(2), pages 298-322.
    11. Lu, Ye & Park, Joon Y., 2019. "Estimation of longrun variance of continuous time stochastic process using discrete sample," Journal of Econometrics, Elsevier, vol. 210(2), pages 236-267.
    12. Hwang, Jungbin & Sun, Yixiao, 2015. "Asymptotic F and t Tests in an Efficient GMM Setting," University of California at San Diego, Economics Working Paper Series qt1c62d8xf, Department of Economics, UC San Diego.
    13. Hwang, Jungbin & Sun, Yixiao, 2017. "Asymptotic F and t tests in an efficient GMM setting," Journal of Econometrics, Elsevier, vol. 198(2), pages 277-295.
    14. Hwang, Jungbin & Sun, Yixiao, 2018. "Should we go one step further? An accurate comparison of one-step and two-step procedures in a generalized method of moments framework," Journal of Econometrics, Elsevier, vol. 207(2), pages 381-405.
    15. Kai Wenger & Christian Leschinski & Philipp Sibbertsen, 2019. "Change-in-mean tests in long-memory time series: a review of recent developments," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 103(2), pages 237-256, June.
    16. Hwang., Jungbin & Sun, Yixiao, 2017. "Simple, Robust, and Accurate F and t Tests in Cointegrated Systems," University of California at San Diego, Economics Working Paper Series qt83b4q8pk, Department of Economics, UC San Diego.
    17. Eben Lazarus & Daniel J. Lewis & James H. Stock, 2021. "The Size‐Power Tradeoff in HAR Inference," Econometrica, Econometric Society, vol. 89(5), pages 2497-2516, September.
    18. Liu, Cheng & Sun, Yixiao, 2019. "A simple and trustworthy asymptotic t test in difference-in-differences regressions," Journal of Econometrics, Elsevier, vol. 210(2), pages 327-362.
    19. Xiaoqing Ye & Yixiao Sun, 2018. "Heteroskedasticity- and autocorrelation-robust F and t tests in Stata," Stata Journal, StataCorp LP, vol. 18(4), pages 951-980, December.
    20. Sun, Yixiao & Yang, Jingjing, 2020. "Testing-optimal kernel choice in HAR inference," Journal of Econometrics, Elsevier, vol. 219(1), pages 123-136.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nya:albaec:13-07. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Byoung Park (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.