IDEAS home Printed from https://ideas.repec.org/a/eee/jbfina/v126y2021ics0378426621000716.html
   My bibliography  Save this article

Data Snooping Bias in Tests of the Relative Performance of Multiple Forecasting Models

Author

Listed:
  • Anghel, Dan Gabriel

Abstract

Tests of the relative performance of multiple forecasting models are sensitive to how the set of alternatives is defined. Evaluating one model against a particular set may show that it has superior predictive ability. However, changing the number or type of alternatives in the set may demonstrate otherwise. This paper focuses on forecasting models based on technical analysis and analyzes how much data snooping bias can occur in tests from restricting the size of forecasting model “universes” or ignoring alternatives used by practitioners and other researchers. A Monte Carlo simulation shows that false discoveries have an average increase of 0.72-2.5 percentage points each time one removes half of the prediction models from the set of relevant alternatives. A complementary empirical investigation suggests that at least 50% of positive findings reported in the literature concerned with trading rule overperformance may be false. Our results motivate several recommendations for applied researchers that would alleviate data snooping bias in some of the more popular statistical tests used in the literature.

Suggested Citation

  • Anghel, Dan Gabriel, 2021. "Data Snooping Bias in Tests of the Relative Performance of Multiple Forecasting Models," Journal of Banking & Finance, Elsevier, vol. 126(C).
  • Handle: RePEc:eee:jbfina:v:126:y:2021:i:c:s0378426621000716
    DOI: 10.1016/j.jbankfin.2021.106113
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378426621000716
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jbankfin.2021.106113?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. Ryan Sullivan & Allan Timmermann & Halbert White, 1999. "Data‐Snooping, Technical Trading Rule Performance, and the Bootstrap," Journal of Finance, American Finance Association, vol. 54(5), pages 1647-1691, October.
    3. Grossman, Sanford J & Stiglitz, Joseph E, 1980. "On the Impossibility of Informationally Efficient Markets," American Economic Review, American Economic Association, vol. 70(3), pages 393-408, June.
    4. Tversky, Amos & Kahneman, Daniel, 1992. "Advances in Prospect Theory: Cumulative Representation of Uncertainty," Journal of Risk and Uncertainty, Springer, vol. 5(4), pages 297-323, October.
    5. Laurent Barras & Olivier Scaillet & Russ Wermers, 2010. "False Discoveries in Mutual Fund Performance: Measuring Luck in Estimated Alphas," Journal of Finance, American Finance Association, vol. 65(1), pages 179-216, February.
    6. Cheol‐Ho Park & Scott H. Irwin, 2007. "What Do We Know About The Profitability Of Technical Analysis?," Journal of Economic Surveys, Wiley Blackwell, vol. 21(4), pages 786-826, September.
    7. Coakley, Jerry & Marzano, Michele & Nankervis, John, 2016. "How profitable are FX technical trading rules?," International Review of Financial Analysis, Elsevier, vol. 45(C), pages 273-282.
    8. Norman R. Swanson & Weiqi Xiong, 2018. "Big data analytics in economics: What have we learned so far, and where should we go from here?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 51(3), pages 695-746, August.
    9. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    10. Bajgrowicz, Pierre & Scaillet, Olivier, 2012. "Technical trading revisited: False discoveries, persistence tests, and transaction costs," Journal of Financial Economics, Elsevier, vol. 106(3), pages 473-491.
    11. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    12. Timmermann, Allan & Granger, Clive W. J., 2004. "Efficient market hypothesis and forecasting," International Journal of Forecasting, Elsevier, vol. 20(1), pages 15-27.
    13. Sánchez-Granero, M.A. & Balladares, K.A. & Ramos-Requena, J.P. & Trinidad-Segovia, J.E., 2020. "Testing the efficient market hypothesis in Latin American stock markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 540(C).
    14. Fama, Eugene F, 1970. "Efficient Capital Markets: A Review of Theory and Empirical Work," Journal of Finance, American Finance Association, vol. 25(2), pages 383-417, May.
    15. Sobreiro, Vinicius Amorim & Cruz Cacique da Costa, Thiago Raymon & Farias Nazário, Rodolfo Toríbio & Lima e Silva, Jéssica & Moreira, Eduardo Alves & Lima Filho, Marcius Correia & Kimura, Herbert & Ar, 2016. "The profitability of moving average trading rules in BRICS and emerging stock markets," The North American Journal of Economics and Finance, Elsevier, vol. 38(C), pages 86-101.
    16. Taylor, Mark P. & Allen, Helen, 1992. "The use of technical analysis in the foreign exchange market," Journal of International Money and Finance, Elsevier, vol. 11(3), pages 304-314, June.
    17. Corbet, Shaen & Eraslan, Veysel & Lucey, Brian & Sensoy, Ahmet, 2019. "The effectiveness of technical trading rules in cryptocurrency markets," Finance Research Letters, Elsevier, vol. 31(C), pages 32-37.
    18. Hsu, Po-Hsuan & Taylor, Mark P. & Wang, Zigan, 2016. "Technical trading: Is it still beating the foreign exchange market?," Journal of International Economics, Elsevier, vol. 102(C), pages 188-208.
    19. Suzanne Fifield & David Power & C. Donald Sinclair, 2005. "An analysis of trading strategies in eleven European stock markets," The European Journal of Finance, Taylor & Francis Journals, vol. 11(6), pages 531-548.
    20. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    21. John D. Storey, 2002. "A direct approach to false discovery rates," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(3), pages 479-498, August.
    22. Tarun Chordia & Amit Goyal & Alessio Saretto & Andrew KarolyiEditor, 2020. "Anomalies and False Rejections," Review of Finance, European Finance Association, vol. 33(5), pages 2134-2179.
    23. Ratner, Mitchell & Leal, Ricardo P. C., 1999. "Tests of technical trading strategies in the emerging equity markets of Latin America and Asia," Journal of Banking & Finance, Elsevier, vol. 23(12), pages 1887-1905, December.
    24. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    25. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    26. Shynkevich, Andrei, 2012. "Performance of technical analysis in growth and small cap segments of the US equity market," Journal of Banking & Finance, Elsevier, vol. 36(1), pages 193-208.
    27. Shynkevich, Andrei, 2016. "Predictability in bond returns using technical trading rules," Journal of Banking & Finance, Elsevier, vol. 70(C), pages 55-69.
    28. Urquhart, Andrew & Gebka, Bartosz & Hudson, Robert, 2015. "How exactly do markets adapt? Evidence from the moving average rule in three developed markets," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 38(C), pages 127-147.
    29. Han, Yufeng & Yang, Ke & Zhou, Guofu, 2013. "A New Anomaly: The Cross-Sectional Profitability of Technical Analysis," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 48(5), pages 1433-1461, October.
    30. Jensen, Michael C., 1978. "Some anomalous evidence regarding market efficiency," Journal of Financial Economics, Elsevier, vol. 6(2-3), pages 95-101.
    31. Hsu, Po-Hsuan & Hsu, Yu-Chin & Kuan, Chung-Ming, 2010. "Testing the predictive ability of technical analysis using a new stepwise test without data snooping bias," Journal of Empirical Finance, Elsevier, vol. 17(3), pages 471-484, June.
    32. Han, Yufeng & Hu, Ting & Yang, Jian, 2016. "Are there exploitable trends in commodity futures prices?," Journal of Banking & Finance, Elsevier, vol. 70(C), pages 214-234.
    33. Shleifer, Andrei & Vishny, Robert W, 1997. "The Limits of Arbitrage," Journal of Finance, American Finance Association, vol. 52(1), pages 35-55, March.
    34. Menkhoff, Lukas, 2010. "The use of technical analysis by fund managers: International evidence," Journal of Banking & Finance, Elsevier, vol. 34(11), pages 2573-2586, November.
    35. Massoud Metghalchi & Juri Marcucci & Yung-Ho Chang, 2012. "Are moving average trading rules profitable? Evidence from the European stock markets," Applied Economics, Taylor & Francis Journals, vol. 44(12), pages 1539-1559, April.
    36. Taylor, Nick, 2014. "The rise and fall of technical trading rule success," Journal of Banking & Finance, Elsevier, vol. 40(C), pages 286-302.
    37. Jin, Sainan & Corradi, Valentina & Swanson, Norman R., 2017. "Robust Forecast Comparison," Econometric Theory, Cambridge University Press, vol. 33(6), pages 1306-1351, December.
    38. Zarrabi, Nima & Snaith, Stuart & Coakley, Jerry, 2017. "FX technical trading rules can be profitable sometimes!," International Review of Financial Analysis, Elsevier, vol. 49(C), pages 113-127.
    39. Hoffmann, Arvid O.I. & Shefrin, Hersh, 2014. "Technical analysis and individual investors," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PB), pages 487-511.
    40. Ben Marshall & Rachael Cahan, 2005. "Is the 52-week high momentum strategy profitable outside the US?," Applied Financial Economics, Taylor & Francis Journals, vol. 15(18), pages 1259-1267.
    41. Tarun Chordia & Amit Goyal & Alessio Saretto, 2020. "Anomalies and False Rejections," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2134-2179.
    42. Kian‐Ping Lim & Robert Brooks, 2011. "The Evolution Of Stock Market Efficiency Over Time: A Survey Of The Empirical Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 25(1), pages 69-108, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anghel, Dan Gabriel, 2022. "No pain, no gain: You should always incorporate trading costs for a bias-free evaluation of trading rule overperformance," Economics Letters, Elsevier, vol. 216(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Psaradellis, Ioannis & Laws, Jason & Pantelous, Athanasios A. & Sermpinis, Georgios, 2023. "Technical analysis, spread trading, and data snooping control," International Journal of Forecasting, Elsevier, vol. 39(1), pages 178-191.
    2. Hung, Chiayu & Lai, Hung-Neng, 2022. "Information asymmetry and the profitability of technical analysis," Journal of Banking & Finance, Elsevier, vol. 134(C).
    3. Hsu, Po-Hsuan & Taylor, Mark P. & Wang, Zigan & Li, Yan, 2024. "The out-of-sample performance of carry trades," Journal of International Money and Finance, Elsevier, vol. 143(C).
    4. Urquhart, Andrew & Zhang, Hanxiong, 2019. "The performance of technical trading rules in Socially Responsible Investments," International Review of Economics & Finance, Elsevier, vol. 63(C), pages 397-411.
    5. Kevin Rink, 2023. "The predictive ability of technical trading rules: an empirical analysis of developed and emerging equity markets," Financial Markets and Portfolio Management, Springer;Swiss Society for Financial Market Research, vol. 37(4), pages 403-456, December.
    6. Robert Hudson & Andrew Urquhart, 2021. "Technical trading and cryptocurrencies," Annals of Operations Research, Springer, vol. 297(1), pages 191-220, February.
    7. Zarrabi, Nima & Snaith, Stuart & Coakley, Jerry, 2017. "FX technical trading rules can be profitable sometimes!," International Review of Financial Analysis, Elsevier, vol. 49(C), pages 113-127.
    8. Urquhart, Andrew & Gebka, Bartosz & Hudson, Robert, 2015. "How exactly do markets adapt? Evidence from the moving average rule in three developed markets," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 38(C), pages 127-147.
    9. Andrei Shynkevich, 2021. "Impact of bitcoin futures on the informational efficiency of bitcoin spot market," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 41(1), pages 115-134, January.
    10. Anghel, Dan Gabriel, 2022. "No pain, no gain: You should always incorporate trading costs for a bias-free evaluation of trading rule overperformance," Economics Letters, Elsevier, vol. 216(C).
    11. ANGHEL, Dan-Gabriel, 2021. "A reality check on trading rule performance in the cryptocurrency market: Machine learning vs. technical analysis," Finance Research Letters, Elsevier, vol. 39(C).
    12. Flavio Ivo Riedlinger & João Nicolau, 2020. "The Profitability in the FTSE 100 Index: A New Markov Chain Approach," Asia-Pacific Financial Markets, Springer;Japanese Association of Financial Economics and Engineering, vol. 27(1), pages 61-81, March.
    13. Vincent, Kendro & Hsu, Yu-Chin & Lin, Hsiou-Wei, 2021. "Investment styles and the multiple testing of cross-sectional stock return predictability," Journal of Financial Markets, Elsevier, vol. 56(C).
    14. Stephen A. Gorman & Frank J. Fabozzi, 2021. "The ABC’s of the alternative risk premium: academic roots," Journal of Asset Management, Palgrave Macmillan, vol. 22(6), pages 405-436, October.
    15. Jin, Xiaoye, 2022. "Performance of intraday technical trading in China’s gold market," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 76(C).
    16. Fong, Tom Pak Wing & Wu, Shui Tang, 2020. "Predictability in sovereign bond returns using technical trading rules: Do developed and emerging markets differ?," The North American Journal of Economics and Finance, Elsevier, vol. 51(C).
    17. Taylor, Mark & Hsu, Po-Hsuan, 2014. "Forty Years, Thirty Currencies and 21,000 Trading Rules: A Large-scale, Data-Snooping Robust Analysis of Technical Trading in t," CEPR Discussion Papers 10018, C.E.P.R. Discussion Papers.
    18. Georgios Sermpinis & Arman Hassanniakalager & Charalampos Stasinakis & Ioannis Psaradellis, 2018. "Technical Analysis and Discrete False Discovery Rate: Evidence from MSCI Indices," Papers 1811.06766, arXiv.org, revised Jun 2019.
    19. Batten, Jonathan A. & Lucey, Brian M. & McGroarty, Frank & Peat, Maurice & Urquhart, Andrew, 2018. "Does intraday technical trading have predictive power in precious metal markets?," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 52(C), pages 102-113.
    20. Xiaoye Jin, 2022. "Evaluating the predictive power of intraday technical trading in China's crude oil market," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(7), pages 1416-1432, November.

    More about this item

    Keywords

    Superior Predictive Ability; Data Snooping Bias; False Discoveries; Reality Check; Technical Analysis; Trading Rules; Efficient Market Hypothesis;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • G11 - Financial Economics - - General Financial Markets - - - Portfolio Choice; Investment Decisions
    • G14 - Financial Economics - - General Financial Markets - - - Information and Market Efficiency; Event Studies; Insider Trading
    • G17 - Financial Economics - - General Financial Markets - - - Financial Forecasting and Simulation

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jbfina:v:126:y:2021:i:c:s0378426621000716. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jbf .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.