IDEAS home Printed from https://ideas.repec.org/a/gam/jrisks/v9y2021i1p18-d476849.html
   My bibliography  Save this article

A Bayesian Approach to Measurement of Backtest Overfitting

Author

Listed:
  • Jiří Witzany

    (Department of Banking and Insurance, Faculty of Finance and Accounting, University of Economics, W. Churchill Sq. 4, 130 67 Prague, Czech Republic)

Abstract

Quantitative investment strategies are often selected from a broad class of candidate models estimated and tested on historical data. Standard statistical techniques to prevent model overfitting such as out-sample backtesting turn out to be unreliable in situations when the selection is based on results of too many models tested on the holdout sample. There is an ongoing discussion of how to estimate the probability of backtest overfitting and adjust the expected performance indicators such as the Sharpe ratio in order to reflect properly the effect of multiple testing. We propose a consistent Bayesian approach that yields the desired robust estimates on the basis of a Markov chain Monte Carlo (MCMC) simulation. The approach is tested on a class of technical trading strategies where a seemingly profitable strategy can be selected in the naïve approach.

Suggested Citation

  • Jiří Witzany, 2021. "A Bayesian Approach to Measurement of Backtest Overfitting," Risks, MDPI, vol. 9(1), pages 1-22, January.
    • Handle: RePEc:gam:jrisks:v:9:y:2021:i:1:p:18-:d:476849
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-9091/9/1/18/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-9091/9/1/18/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    2. Marcos López de Prado & Michael J. Lewis, 2019. "Detection of false investment strategies using unsupervised learning methods," Quantitative Finance, Taylor & Francis Journals, vol. 19(9), pages 1555-1565, September.
    3. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    4. Andrew Y Chen & Tom Zimmermann & Jeffrey Pontiff, 2020. "Publication Bias and the Cross-Section of Stock Returns," The Review of Asset Pricing Studies, Society for Financial Studies, vol. 10(2), pages 249-289.
    5. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chinco, Alex & Neuhierl, Andreas & Weber, Michael, 2021. "Estimating the anomaly base rate," Journal of Financial Economics, Elsevier, vol. 140(1), pages 101-126.
    2. Andrew Y. Chen & Tom Zimmermann, 2022. "Publication Bias in Asset Pricing Research," Papers 2209.13623, arXiv.org, revised Sep 2023.
    3. Andrew Y. Chen, 2021. "The Limits of p‐Hacking: Some Thought Experiments," Journal of Finance, American Finance Association, vol. 76(5), pages 2447-2480, October.
    4. Andrew Y. Chen, 2022. "Do t-Statistic Hurdles Need to be Raised?," Papers 2204.10275, arXiv.org, revised Apr 2024.
    5. Heath, Davidson & Ringgenberg, Matthew C. & Samadi, Mehrdad & Werner, Ingrid M., 2019. "Reusing Natural Experiments," Working Paper Series 2019-21, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    6. Todd Mitton, 2022. "Methodological Variation in Empirical Corporate Finance," The Review of Financial Studies, Society for Financial Studies, vol. 35(2), pages 527-575.
    7. Minyou Fan & Youwei Li & Ming Liao & Jiadong Liu, 2022. "A reexamination of factor momentum: How strong is it?," The Financial Review, Eastern Finance Association, vol. 57(3), pages 585-615, August.
    8. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    9. Andrew Y. Chen & Chukwuma Dim, 2023. "High-Throughput Asset Pricing," Papers 2311.10685, arXiv.org, revised Jun 2025.
    10. Andrew Y. Chen, 2022. "Most claimed statistical findings in cross-sectional return predictability are likely true," Papers 2206.15365, arXiv.org, revised Jun 2025.
    11. Benjamin R. Auer, 2022. "On false discoveries of standard t-tests in investment management applications," Review of Managerial Science, Springer, vol. 16(3), pages 751-768, April.
    12. Harvey, Campbell R. & Liu, Yan, 2021. "Lucky factors," Journal of Financial Economics, Elsevier, vol. 141(2), pages 413-435.
    13. Marcelo Fernandes & Breno Neri, 2010. "Nonparametric Entropy-Based Tests of Independence Between Stochastic Processes," Econometric Reviews, Taylor & Francis Journals, vol. 29(3), pages 276-306.
    14. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    15. Clark, Todd E. & McCracken, Michael W., 2012. "In-sample tests of predictive ability: A new approach," Journal of Econometrics, Elsevier, vol. 170(1), pages 1-14.
    16. Sager, Michael & Taylor, Mark P., 2014. "Generating currency trading rules from the term structure of forward foreign exchange premia," Journal of International Money and Finance, Elsevier, vol. 44(C), pages 230-250.
    17. Marco Rocco, 2011. "Extreme value theory for finance: a survey," Questioni di Economia e Finanza (Occasional Papers) 99, Bank of Italy, Economic Research and International Relations Area.
    18. Corradi, Valentina & Swanson, Norman R., 2004. "A test for the distributional comparison of simulated and historical data," Economics Letters, Elsevier, vol. 85(2), pages 185-193, November.
    19. Todd E. Clark & Michael W. McCracken, 2010. "Averaging forecasts from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(1), pages 5-29, January.
    20. Castle Jennifer L. & Doornik Jurgen A & Hendry David F., 2011. "Evaluating Automatic Model Selection," Journal of Time Series Econometrics, De Gruyter, vol. 3(1), pages 1-33, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jrisks:v:9:y:2021:i:1:p:18-:d:476849. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.