IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v38y2022i3p872-877.html

In-sample tests of predictability are superior to pseudo-out-of-sample tests, even when data mining

Author

Listed:
  • Hunt, Ian

Abstract

This paper analyses straightforward Bonferroni adjustments to critical values of in-sample tests of predictability, when data mining is used to search across models. Unlike conventional pseudo-out-of-sample tests, these in-sample tests have stable family-wise error rates (FWERs) when searching for models that predict well. Furthermore, when data mining, these in-sample tests have more power than pseudo-out-of-sample tests for identifying true predictability.

Suggested Citation

  • Hunt, Ian, 2022. "In-sample tests of predictability are superior to pseudo-out-of-sample tests, even when data mining," International Journal of Forecasting, Elsevier, vol. 38(3), pages 872-877.
    • Handle: RePEc:eee:intfor:v:38:y:2022:i:3:p:872-877
    DOI: 10.1016/j.ijforecast.2021.05.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207021000789
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2021.05.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Peter Reinhard Hansen & Allan Timmermann, 2015. "Equivalence Between Out‐of‐Sample Forecast Comparisons and Wald Statistics," Econometrica, Econometric Society, vol. 83, pages 2485-2505, November.
    2. Francis X. Diebold, 2015. "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(1), pages 1-1, January.
    3. Yoav Benjamini & Marina Bogomolov, 2014. "Selective inference on multiple families of hypotheses," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 297-318, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paul MUKUCHA & Divaries Cosmas JARAVAZA & Fanny SARUCHERA, 2025. "Strategic Postponement In Fast Food Operations: Enhancing Order Fulfilment In A Frontier Emerging Market," Business Excellence and Management, Faculty of Management, Academy of Economic Studies, Bucharest, Romania, vol. 15(2), pages 5-18, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rui Liu, 2019. "Forecasting Bond Risk Premia with Unspanned Macroeconomic Information," Quarterly Journal of Finance (QJF), World Scientific Publishing Co. Pte. Ltd., vol. 9(01), pages 1-62, March.
    2. Alessandro Morico & Ovidijus Stauskas, 2025. "Robust Tests for Factor-Augmented Regressions with an Application to the novel EA-MD-QD Dataset," Papers 2504.08455, arXiv.org, revised Nov 2025.
    3. Pitarakis, Jean-Yves, 2025. "A Novel Approach To Predictive Accuracy Testing In Nested Environments," Econometric Theory, Cambridge University Press, vol. 41(1), pages 35-78, February.
    4. Knut Are Aastveit & Claudia Foroni & Francesco Ravazzolo, 2017. "Density Forecasts With Midas Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(4), pages 783-801, June.
    5. Florian Ziel & Rick Steinert & Sven Husmann, 2015. "Forecasting day ahead electricity spot prices: The impact of the EXAA to other European electricity markets," Papers 1501.00818, arXiv.org, revised Dec 2015.
    6. Ellwanger, Reinhard & Snudden, Stephen, 2025. "Putting VAR forecasts of the real price of crude oil to the test," Finance Research Letters, Elsevier, vol. 77(C).
    7. Seitz, Franz & Baumann, Ursel & Albuquerque, Bruno, 2015. "The information content of money and credit for US activity," Working Paper Series 1803, European Central Bank.
    8. Philippe Goulet Coulombe & Maxime Leroux & Dalibor Stevanovic & Stéphane Surprenant, 2022. "How is machine learning useful for macroeconomic forecasting?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 920-964, August.
    9. Warne, Anders, 2023. "DSGE model forecasting: rational expectations vs. adaptive learning," Working Paper Series 2768, European Central Bank.
    10. Christopher Kath & Florian Ziel, 2018. "The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts," Papers 1811.08604, arXiv.org.
    11. Joseph P. Byrne & Dimitris Korobilis & Pinho J. Ribeiro, 2018. "On The Sources Of Uncertainty In Exchange Rate Predictability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 59(1), pages 329-357, February.
    12. Ding, Song & Cai, Zhijian & Qin, Xinghuan & Shen, Xingao, 2024. "Comparative assessment and policy analysis of forecasting quarterly renewable energy demand: Fresh evidence from an innovative seasonal approach with superior matching algorithms," Applied Energy, Elsevier, vol. 367(C).
    13. Simon Hirsch, 2025. "Online Multivariate Regularized Distributional Regression for High-dimensional Probabilistic Electricity Price Forecasting," Papers 2504.02518, arXiv.org, revised Oct 2025.
    14. Rodney Edvinsson & Sune Karlsson & Pär Österholm, 2025. "Does money growth predict inflation in Sweden? Evidence from vector autoregressions using four centuries of data," Empirical Economics, Springer, vol. 68(4), pages 1613-1635, April.
    15. Catherine L. Kling & Raymond W. Arritt & Gray Calhoun & David A. Keiser, 2016. "Research Needs and Challenges in the FEW System: Coupling Economic Models with Agronomic, Hydrologic, and Bioenergy Models for Sustainable Food, Energy, and Water Systems," Center for Agricultural and Rural Development (CARD) Publications 16-wp563, Center for Agricultural and Rural Development (CARD) at Iowa State University.
    16. Karen Poghosyan & Ruben Poghosyan, 2021. "On the Applicability of Dynamic Factor Models for Forecasting Real GDP Growth in Armenia," Czech Journal of Economics and Finance (Finance a uver), Charles University Prague, Faculty of Social Sciences, vol. 71(1), pages 52-79, June.
    17. Xu, Hua & Wang, Minggang & Jiang, Shumin & Yang, Weiguo, 2020. "Carbon price forecasting with complex network and extreme learning machine," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 545(C).
    18. David T. Frazier & Donald S. Poskitt, 2025. "Sequential Scoring Rule Evaluation for Forecast Method Selection," Papers 2505.09090, arXiv.org.
    19. Papantonis, Ioannis & Rompolis, Leonidas & Tzavalis, Elias, 2023. "Improving variance forecasts: The role of Realized Variance features," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1221-1237.
    20. Mai Dao & Lam Nguyen, 2025. "Variable selection in macroeconomic stress test: a Bayesian quantile regression approach," Empirical Economics, Springer, vol. 68(3), pages 1113-1169, March.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:38:y:2022:i:3:p:872-877. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.