IDEAS home Printed from https://ideas.repec.org/a/bla/jfinan/v79y2024i1p459-503.html

The Virtue of Complexity in Return Prediction

Author

Listed:
  • BRYAN KELLY
  • SEMYON MALAMUD
  • KANGYING ZHOU

Abstract

Much of the extant literature predicts market returns with “simple” models that use only a few parameters. Contrary to conventional wisdom, we theoretically prove that simple models severely understate return predictability compared to “complex” models in which the number of parameters exceeds the number of observations. We empirically document the virtue of complexity in U.S. equity market return prediction. Our findings establish the rationale for modeling expected returns through machine learning.

Suggested Citation

  • Bryan Kelly & Semyon Malamud & Kangying Zhou, 2024. "The Virtue of Complexity in Return Prediction," Journal of Finance, American Finance Association, vol. 79(1), pages 459-503, February.
    • Handle: RePEc:bla:jfinan:v:79:y:2024:i:1:p:459-503
    DOI: 10.1111/jofi.13298
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/jofi.13298
    Download Restriction: no
    File URL: https://libkey.io/10.1111/jofi.13298?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hansen, Lars Peter & Richard, Scott F, 1987. "The Role of Conditioning Information in Deducing Testable," Econometrica, Econometric Society, vol. 55(3), pages 587-613, May.
    2. Cenesizoglu, Tolga & Timmermann, Allan, 2012. "Do return prediction models add economic value?," Journal of Banking & Finance, Elsevier, vol. 36(11), pages 2974-2987.
    3. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    4. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    5. Silverstein, J. W., 1995. "Strong Convergence of the Empirical Distribution of Eigenvalues of Large Dimensional Random Matrices," Journal of Multivariate Analysis, Elsevier, vol. 55(2), pages 331-339, November.
    6. Xi Dong & Yan Li & David E. Rapach & Guofu Zhou, 2022. "Anomalies and the Expected Market Return," Journal of Finance, American Finance Association, vol. 77(1), pages 639-681, February.
    7. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    8. Domenico Giannone & Michele Lenza & Giorgio E. Primiceri, 2021. "Economic Predictions With Big Data: The Illusion of Sparsity," Econometrica, Econometric Society, vol. 89(5), pages 2409-2437, September.
    9. John H. Cochrane, 2011. "Presidential Address: Discount Rates," Journal of Finance, American Finance Association, vol. 66(4), pages 1047-1108, August.
    10. Martin, Ian W.R. & Nagel, Stefan, 2022. "Market efficiency in the age of big data," Journal of Financial Economics, Elsevier, vol. 145(1), pages 154-177.
    11. Joachim Freyberger & Andreas Neuhierl & Michael Weber & Andrew KarolyiEditor, 2020. "Dissecting Characteristics Nonparametrically," Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2326-2377.
    12. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    13. Abhyankar, Abhay & Basu, Devraj & Stremme, Alexander, 2012. "The Optimal Use of Return Predictability: An Empirical Study," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 47(5), pages 973-1001, October.
    14. Jianqing Fan & Jianhua Guo & Shurong Zheng, 2022. "Estimating Number of Factors by Adjusted Eigenvalues Thresholding," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(538), pages 852-861, April.
    15. David E. Rapach & Jack K. Strauss & Guofu Zhou, 2010. "Out-of-Sample Equity Premium Prediction: Combination Forecasts and Links to the Real Economy," The Review of Financial Studies, Society for Financial Studies, vol. 23(2), pages 821-862, February.
    16. Silverstein, J. W. & Bai, Z. D., 1995. "On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices," Journal of Multivariate Analysis, Elsevier, vol. 54(2), pages 175-192, August.
    17. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-590, June.
    18. Fan, Jianqing & Fan, Yingying & Lv, Jinchi, 2008. "High dimensional covariance matrix estimation using a factor model," Journal of Econometrics, Elsevier, vol. 147(1), pages 186-197, November.
    19. Wayne E. Ferson & Andrew F. Siegel, 2001. "The Efficient Use of Conditioning Information in Portfolios," Journal of Finance, American Finance Association, vol. 56(3), pages 967-982, June.
    20. Ralph S.J. Koijen & Stijn Van Nieuwerburgh, 2011. "Predictability of Returns and Cash Flows," Annual Review of Financial Economics, Annual Reviews, vol. 3(1), pages 467-491, December.
    21. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuan, Ying & Qu, Yong & Wang, Tianyang, 2025. "Predicting risk premiums: A constraint-based model," Journal of Empirical Finance, Elsevier, vol. 83(C).
    2. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    3. Fabrizio Ghezzi & Anindo Sarkar & Thomas Quistgaard Pedersen & Allan Timmermann, 2025. "Optimal asset allocation and nonlinear return predictability from the dividend-price ratio," Annals of Operations Research, Springer, vol. 346(1), pages 415-445, March.
    4. Xi Dong & Yan Li & David E. Rapach & Guofu Zhou, 2022. "Anomalies and the Expected Market Return," Journal of Finance, American Finance Association, vol. 77(1), pages 639-681, February.
    5. Christian Fieberg & Daniel Metko & Thorsten Poddig & Thomas Loy, 2023. "Machine learning techniques for cross-sectional equity returns’ prediction," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 45(1), pages 289-323, March.
    6. Smith, Simon C., 2021. "International stock return predictability," International Review of Financial Analysis, Elsevier, vol. 78(C).
    7. Maung, Kenwin & Swanson, Norman R., 2025. "A survey of models and methods used for forecasting when investing in financial markets," International Journal of Forecasting, Elsevier, vol. 41(4), pages 1355-1382.
    8. Lof, Matthijs & Nyberg, Henri, 2024. "Discount rates and cash flows: A local projection approach," Journal of Banking & Finance, Elsevier, vol. 162(C).
    9. Angelidis, Timotheos & Sakkas, Athanasios & Tessaromatis, Nikolaos, 2025. "Predicting commodity returns: Time series vs. cross sectional prediction models," Journal of Commodity Markets, Elsevier, vol. 38(C).
    10. Babiak, Mykola & Baruník, Jozef, 2026. "Deep learning, predictability, and optimal portfolio returns," Journal of Empirical Finance, Elsevier, vol. 87(C).
    11. Lawrenz, Jochen & Zorn, Josef, 2017. "Predicting international stock returns with conditional price-to-fundamental ratios," Journal of Empirical Finance, Elsevier, vol. 43(C), pages 159-184.
    12. Giovannelli, Alessandro & Massacci, Daniele & Soccorsi, Stefano, 2021. "Forecasting stock returns with large dimensional factor models," Journal of Empirical Finance, Elsevier, vol. 63(C), pages 252-269.
    13. Wang, Jianqiu & Wang, Zhuo & Wu, Ke, 2025. "Forecasting stock market return with anomalies: Evidence from China," International Journal of Forecasting, Elsevier, vol. 41(3), pages 1278-1295.
    14. Díaz, Juan D. & Hansen, Erwin & Cabrera, Gabriel, 2023. "Gold risk premium estimation with machine learning methods," Journal of Commodity Markets, Elsevier, vol. 31(C).
    15. Yaojie Zhang & Mengxi He & Zhikai Zhang, 2024. "Forecasting stock returns with industry volatility concentration," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(7), pages 2705-2730, November.
    16. Fabian Hollstein & Marcel Prokopczuk, 2023. "Managing the Market Portfolio," Management Science, INFORMS, vol. 69(6), pages 3675-3696, June.
    17. Daniel de Almeida & Ana-Maria Fuertes & Luiz Koodi Hotta, 2025. "Out-of-Sample Predictability of the Equity Risk Premium," Mathematics, MDPI, vol. 13(2), pages 1-23, January.
    18. Zhao, Albert Bo & Cheng, Tingting, 2022. "Stock return prediction: Stacking a variety of models," Journal of Empirical Finance, Elsevier, vol. 67(C), pages 288-317.
    19. Moench, Emanuel & Stein, Tobias, 2026. "Equity Premium Predictability over the Business Cycle," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 61(3), pages 1216-1246, May.
    20. Boudoukh, Jacob & Israel, Ronen & Richardson, Matthew, 2022. "Biases in long-horizon predictive regressions," Journal of Financial Economics, Elsevier, vol. 145(3), pages 937-969.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jfinan:v:79:y:2024:i:1:p:459-503. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/afaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.