IDEAS home Printed from
   My bibliography  Save this paper

Backtesting Strategies Based on Multiple Signals


  • Robert Novy-Marx


Strategies selected by combining multiple signals suffer severe overfitting biases, because underlying signals are typically signed such that each predicts positive in-sample returns. “Highly significant” backtested performance is easy to generate by selecting stocks on the basis of combinations of randomly generated signals, which by construction have no true power. This paper analyzes t-statistic distributions for multi-signal strategies, both empirically and theoretically, to determine appropriate critical values, which can be several times standard levels. Overfitting bias also severely exacerbates the multiple testing bias that arises when investigators consider more results than they present. Combining the best k out of n candidate signals yields a bias almost as large as those obtained by selecting the single best of nᵏ candidate signals.

Suggested Citation

  • Robert Novy-Marx, 2015. "Backtesting Strategies Based on Multiple Signals," NBER Working Papers 21329, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:21329
    Note: AP

    Download full text from publisher

    File URL:
    Download Restriction: no

    References listed on IDEAS

    1. Paul Gompers & Joy Ishii & Andrew Metrick, 2003. "Corporate Governance and Equity Prices," The Quarterly Journal of Economics, Oxford University Press, vol. 118(1), pages 107-156.
    2. Piotroski, JD, 2000. "Value investing: The use of historical financial statement information to separate winners from losers," Journal of Accounting Research, Wiley Blackwell, vol. 38, pages 1-41.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Eduard Krkoska & Klaus Reiner Schenk-Hoppé, 2019. "Herding in Smart-Beta Investment Products," JRFM, MDPI, vol. 12(1), pages 1-14, March.
    2. Dirk Paulsen & Jakob Sohl, 2016. "Noise Fit, Estimation Error and a Sharpe Information Criterion," Papers 1602.06186,, revised Dec 2019.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kewei Hou & Haitao Mo & Chen Xue & Lu Zhang, 2019. "Which Factors?," Review of Finance, European Finance Association, vol. 23(1), pages 1-35.
    2. Hou, Kewei & Xue, Chen & Zhang, Lu, 2017. "Replicating Anomalies," Working Paper Series 2017-10, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    3. Efrat Shust & Dan Weiss, 2022. "Disentangling Relevance from Reliability in Value Relevance Tests," Sustainability, MDPI, vol. 14(20), pages 1-15, October.
    4. Jonathan A. Christy & Zoltan P. Matolcsy & Anna Wright & Anne Wyatt, 2013. "Do Board Characteristics Influence the Shareholders' Assessment of Risk for Small and Large Firms?," Abacus, Accounting Foundation, University of Sydney, vol. 49(2), pages 161-196, June.
    5. Kelvin K. F. Law & Lillian F. Mills, 2015. "Taxes and Financial Constraints: Evidence from Linguistic Cues," Journal of Accounting Research, Wiley Blackwell, vol. 53(4), pages 777-819, September.
    6. Manuel Ammann & Philipp Horsch & David Oesch, 2016. "Competing with Superstars," Management Science, INFORMS, vol. 62(10), pages 2842-2858, October.
    7. Enikolopov, Ruben & Petrova, Maria & Stepanov, Sergey, 2014. "Firm value in crisis: Effects of firm-level transparency and country-level institutions," Journal of Banking & Finance, Elsevier, vol. 46(C), pages 72-84.
    8. Kim, Sang-Joon & Bae, John & Oh, Hannah, 2019. "Financing strategically: The moderation effect of marketing activities on the bifurcated relationship between debt level and firm valuation of small and medium enterprises," The North American Journal of Economics and Finance, Elsevier, vol. 48(C), pages 663-681.
    9. Smith, Deborah Drummond & Gleason, Kimberly C. & Kannan, Yezen H., 2021. "Auditor liability and excess cash holdings: Evidence from audit fees of foreign incorporated firms," International Review of Financial Analysis, Elsevier, vol. 78(C).
    10. Custódio, Cláudia & Ferreira, Miguel A. & Laureano, Luís, 2013. "Why are US firms using more short-term debt?," Journal of Financial Economics, Elsevier, vol. 108(1), pages 182-212.
    11. Qiu, Buhui & Trapkov, Svetoslav & Yakoub, Fadi, 2014. "Do target CEOs trade premiums for personal benefits?," Journal of Banking & Finance, Elsevier, vol. 42(C), pages 23-41.
    12. Bae, John & Kim, Sang-Joon & Oh, Hannah, 2017. "Taming polysemous signals: The role of marketing intensity on the relationship between financial leverage and firm performance," Review of Financial Economics, Elsevier, vol. 33(C), pages 29-40.
    13. Francesca Franco & Christopher D. Ittner & Oktay Urcan, 2017. "Determinants and Trading Performance of Equity Deferrals by Corporate Outside Directors," Management Science, INFORMS, vol. 63(1), pages 114-138, January.
    14. Ng, Joe Cho Yiu & Leung, Charles Ka Yui & Chan, Suikang, 2022. "Corporate Real Estate Holding and Stock Returns: International Evidence from Listed Companies," MPRA Paper 111691, University Library of Munich, Germany.
    15. Al-Amri, Khalid & Davydov, Yevgeniy, 2016. "Testing the effectiveness of ERM: Evidence from operational losses," Journal of Economics and Business, Elsevier, vol. 87(C), pages 70-82.
    16. Renneboog, L.D.R. & Szilagyi, P.G., 2009. "Shareholder Activism through the Proxy Process," Other publications TiSEM cc25d736-2965-4511-b100-1, Tilburg University, School of Economics and Management.
    17. Francis, Bill & Hasan, Iftekhar & Sharma, Zenu, 2011. "Leverage and growth: Effect of stock options," Journal of Economics and Business, Elsevier, vol. 63(6), pages 558-581.
    18. Keming Li, 2021. "The effect of option trading," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 7(1), pages 1-32, December.
    19. Manhwa Wu & Paoyu Huang & Yensen Ni, 2020. "The Impact of Institutional Shareholdings on Price Limits," Asia-Pacific Financial Markets, Springer;Japanese Association of Financial Economics and Engineering, vol. 27(3), pages 343-361, September.
    20. Tong, Hui & Wei, Shang-Jin, 2014. "Does trade globalization induce or inhibit corporate transparency? Unbundling the growth potential and product market competition channels," Journal of International Economics, Elsevier, vol. 94(2), pages 358-370.

    More about this item

    JEL classification:

    • C58 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Financial Econometrics
    • G11 - Financial Economics - - General Financial Markets - - - Portfolio Choice; Investment Decisions

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:21329. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.