IDEAS home Printed from https://ideas.repec.org/a/bla/jfinan/v79y2024i1p459-503.html

The Virtue of Complexity in Return Prediction

Author

Listed:
  • BRYAN KELLY
  • SEMYON MALAMUD
  • KANGYING ZHOU

Abstract

Much of the extant literature predicts market returns with “simple” models that use only a few parameters. Contrary to conventional wisdom, we theoretically prove that simple models severely understate return predictability compared to “complex” models in which the number of parameters exceeds the number of observations. We empirically document the virtue of complexity in U.S. equity market return prediction. Our findings establish the rationale for modeling expected returns through machine learning.

Suggested Citation

  • Bryan Kelly & Semyon Malamud & Kangying Zhou, 2024. "The Virtue of Complexity in Return Prediction," Journal of Finance, American Finance Association, vol. 79(1), pages 459-503, February.
    • Handle: RePEc:bla:jfinan:v:79:y:2024:i:1:p:459-503
    DOI: 10.1111/jofi.13298
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/jofi.13298
    Download Restriction: no
    File URL: https://libkey.io/10.1111/jofi.13298?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    2. Abhyankar, Abhay & Basu, Devraj & Stremme, Alexander, 2012. "The Optimal Use of Return Predictability: An Empirical Study," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 47(5), pages 973-1001, October.
    3. Hansen, Lars Peter & Richard, Scott F, 1987. "The Role of Conditioning Information in Deducing Testable," Econometrica, Econometric Society, vol. 55(3), pages 587-613, May.
    4. Jianqing Fan & Jianhua Guo & Shurong Zheng, 2022. "Estimating Number of Factors by Adjusted Eigenvalues Thresholding," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(538), pages 852-861, April.
    5. Cenesizoglu, Tolga & Timmermann, Allan, 2012. "Do return prediction models add economic value?," Journal of Banking & Finance, Elsevier, vol. 36(11), pages 2974-2987.
    6. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    7. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    8. Silverstein, J. W., 1995. "Strong Convergence of the Empirical Distribution of Eigenvalues of Large Dimensional Random Matrices," Journal of Multivariate Analysis, Elsevier, vol. 55(2), pages 331-339, November.
    9. Martin, Ian W.R. & Nagel, Stefan, 2022. "Market efficiency in the age of big data," Journal of Financial Economics, Elsevier, vol. 145(1), pages 154-177.
    10. David E. Rapach & Jack K. Strauss & Guofu Zhou, 2010. "Out-of-Sample Equity Premium Prediction: Combination Forecasts and Links to the Real Economy," The Review of Financial Studies, Society for Financial Studies, vol. 23(2), pages 821-862, February.
    11. Joachim Freyberger & Andreas Neuhierl & Michael Weber, 2020. "Dissecting Characteristics Nonparametrically," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2326-2377.
    12. Xi Dong & Yan Li & David E. Rapach & Guofu Zhou, 2022. "Anomalies and the Expected Market Return," Journal of Finance, American Finance Association, vol. 77(1), pages 639-681, February.
    13. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    14. Domenico Giannone & Michele Lenza & Giorgio E. Primiceri, 2021. "Economic Predictions With Big Data: The Illusion of Sparsity," Econometrica, Econometric Society, vol. 89(5), pages 2409-2437, September.
    15. Silverstein, J. W. & Bai, Z. D., 1995. "On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices," Journal of Multivariate Analysis, Elsevier, vol. 54(2), pages 175-192, August.
    16. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-590, June.
    17. John H. Cochrane, 2011. "Presidential Address: Discount Rates," Journal of Finance, American Finance Association, vol. 66(4), pages 1047-1108, August.
    18. Fan, Jianqing & Fan, Yingying & Lv, Jinchi, 2008. "High dimensional covariance matrix estimation using a factor model," Journal of Econometrics, Elsevier, vol. 147(1), pages 186-197, November.
    19. Wayne E. Ferson & Andrew F. Siegel, 2001. "The Efficient Use of Conditioning Information in Portfolios," Journal of Finance, American Finance Association, vol. 56(3), pages 967-982, June.
    20. Ralph S.J. Koijen & Stijn Van Nieuwerburgh, 2011. "Predictability of Returns and Cash Flows," Annual Review of Financial Economics, Annual Reviews, vol. 3(1), pages 467-491, December.
    21. Joachim Freyberger & Andreas Neuhierl & Michael Weber & Andrew KarolyiEditor, 2020. "Dissecting Characteristics Nonparametrically," Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2326-2377.
    22. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Fabrizio Ghezzi & Anindo Sarkar & Thomas Quistgaard Pedersen & Allan Timmermann, 2025. "Optimal asset allocation and nonlinear return predictability from the dividend-price ratio," Annals of Operations Research, Springer, vol. 346(1), pages 415-445, March.
    3. Xi Dong & Yan Li & David E. Rapach & Guofu Zhou, 2022. "Anomalies and the Expected Market Return," Journal of Finance, American Finance Association, vol. 77(1), pages 639-681, February.
    4. Angelidis, Timotheos & Sakkas, Athanasios & Tessaromatis, Nikolaos, 2025. "Predicting commodity returns: Time series vs. cross sectional prediction models," Journal of Commodity Markets, Elsevier, vol. 38(C).
    5. Yuan, Ying & Qu, Yong & Wang, Tianyang, 2025. "Predicting risk premiums: A constraint-based model," Journal of Empirical Finance, Elsevier, vol. 83(C).
    6. Babiak, Mykola & Baruník, Jozef, 2026. "Deep learning, predictability, and optimal portfolio returns," Journal of Empirical Finance, Elsevier, vol. 87(C).
    7. Wang, Jianqiu & Wang, Zhuo & Wu, Ke, 2025. "Forecasting stock market return with anomalies: Evidence from China," International Journal of Forecasting, Elsevier, vol. 41(3), pages 1278-1295.
    8. Christian Fieberg & Daniel Metko & Thorsten Poddig & Thomas Loy, 2023. "Machine learning techniques for cross-sectional equity returns’ prediction," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 45(1), pages 289-323, March.
    9. Smith, Simon C., 2021. "International stock return predictability," International Review of Financial Analysis, Elsevier, vol. 78(C).
    10. Maung, Kenwin & Swanson, Norman R., 2025. "A survey of models and methods used for forecasting when investing in financial markets," International Journal of Forecasting, Elsevier, vol. 41(4), pages 1355-1382.
    11. Cakici, Nusret & Fieberg, Christian & Metko, Daniel & Zaremba, Adam, 2023. "Machine learning goes global: Cross-sectional return predictability in international stock markets," Journal of Economic Dynamics and Control, Elsevier, vol. 155(C).
    12. Lof, Matthijs & Nyberg, Henri, 2024. "Discount rates and cash flows: A local projection approach," Journal of Banking & Finance, Elsevier, vol. 162(C).
    13. Huang, Dashan & Li, Jiangyuan & Wang, Liyao, 2021. "Are disagreements agreeable? Evidence from information aggregation," Journal of Financial Economics, Elsevier, vol. 141(1), pages 83-101.
    14. Dichtl, Hubert & Drobetz, Wolfgang & Otto, Tizian, 2023. "Forecasting Stock Market Crashes via Machine Learning," Journal of Financial Stability, Elsevier, vol. 65(C).
    15. Penaranda, Francisco & Sentana, Enrique, 2024. "Portfolio management with big data," CEPR Discussion Papers 19314, C.E.P.R. Discussion Papers.
    16. Wolfgang Drobetz & Tizian Otto, 2021. "Empirical asset pricing via machine learning: evidence from the European stock market," Journal of Asset Management, Palgrave Macmillan, vol. 22(7), pages 507-538, December.
    17. Pan, Zhiyuan & Zhong, Hao & Wang, Yudong & Huang, Juan, 2024. "Forecasting oil futures returns with news," Energy Economics, Elsevier, vol. 134(C).
    18. Lawrenz, Jochen & Zorn, Josef, 2017. "Predicting international stock returns with conditional price-to-fundamental ratios," Journal of Empirical Finance, Elsevier, vol. 43(C), pages 159-184.
    19. Giovannelli, Alessandro & Massacci, Daniele & Soccorsi, Stefano, 2021. "Forecasting stock returns with large dimensional factor models," Journal of Empirical Finance, Elsevier, vol. 63(C), pages 252-269.
    20. Wu, Haoran & Gao, Zhiwei & Nie, Boyang & Zhao, Binru, 2025. "Can machines learn Chinese mutual funds?," Pacific-Basin Finance Journal, Elsevier, vol. 94(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jfinan:v:79:y:2024:i:1:p:459-503. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/afaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.