IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2601.07992.html

Fake Date Tests: Can We Trust In-sample Accuracy of LLMs in Macroeconomic Forecasting?

Author

Listed:
  • Alexander Eliseev
  • Sergei Seleznev

Abstract

Large language models (LLMs) are a type of machine learning tool that economists have started to apply in their empirical research. One such application is macroeconomic forecasting with backtesting of LLMs, even though they are trained on the same data that is used to estimate their forecasting performance. Can these in-sample accuracy results be extrapolated to the model's out-of-sample performance? To answer this question, we developed a family of prompt sensitivity tests and two members of this family, which we call the fake date tests. These tests aim to detect two types of biases in LLMs' in-sample forecasts: lookahead bias and context bias. According to the empirical results, none of the modern LLMs tested in this study passed our first test, signaling the presence of lookahead bias in their in-sample forecasts.

Suggested Citation

  • Alexander Eliseev & Sergei Seleznev, 2026. "Fake Date Tests: Can We Trust In-sample Accuracy of LLMs in Macroeconomic Forecasting?," Papers 2601.07992, arXiv.org, revised Mar 2026.
    • Handle: RePEc:arx:papers:2601.07992
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2601.07992
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Daron Acemoglu, 2025. "The simple macroeconomics of AI," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 40(121), pages 13-58.
    2. Diebold, Francis X. & Schorfheide, Frank & Shin, Minchul, 2017. "Real-time forecast evaluation of DSGE models with stochastic volatility," Journal of Econometrics, Elsevier, vol. 201(2), pages 322-332.
    3. Alejandro Lopez-Lira & Yuehua Tang & Mingyin Zhu, 2025. "The Memorization Problem: Can We Trust LLMs' Economic Forecasts?," Papers 2504.14765, arXiv.org, revised Dec 2025.
    4. Erik Brynjolfsson & Anton Korinek & Ajay K. Agrawal, 2025. "A Research Agenda for the Economics of Transformative AI," NBER Working Papers 34256, National Bureau of Economic Research, Inc.
    5. Miguel Faria-e-Castro & Fernando Leibovici, 2024. "Artificial Intelligence and Inflation Forecasts," Review, Federal Reserve Bank of St. Louis, vol. 106(12), pages 1-14, November.
    6. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org, revised Feb 2025.
    7. Ajay K. Agrawal & Erik Brynjolfsson & Anton Korinek, 2025. "Introduction to "The Economics of Transformative AI"," NBER Chapters, in: The Economics of Transformative AI, pages 1-11, National Bureau of Economic Research, Inc.
    8. Zarifhonarvar, Ali, 2026. "Generating inflation expectations with large language models," Journal of Monetary Economics, Elsevier, vol. 157(C).
    9. Jens Ludwig & Sendhil Mullainathan & Ashesh Rambachan, 2026. "Large Language Models: An Applied Econometric Framework," Annual Review of Economics, Annual Reviews, vol. 18(1), pages 283-316, August.
    10. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    11. Domenico Giannone & Michele Lenza & Giorgio E. Primiceri, 2015. "Prior Selection for Vector Autoregressions," The Review of Economics and Statistics, MIT Press, vol. 97(2), pages 436-451, May.
    12. Sophia Kazinnik & Tara M. Sinclair, 2025. "FOMC In Silico: A Multi-Agent System for Monetary Policy Decision Modeling," Working Papers 2025-005, The George Washington University, The Center for Economic Research.
    13. Jianhao Lin & Lexuan Sun & Yixin Yan, 2025. "Simulating Macroeconomic Expectations in Survey Experiments with LLM-based Economic Agents," Papers 2505.17648, arXiv.org, revised Jun 2026.
    14. Didisheim, Antoine & Fraschini, Martina & Somoza, Luciano, 2025. "AI’s predictable memory in financial analysis," Economics Letters, Elsevier, vol. 256(C).
    15. Songrun He & Linying Lv & Asaf Manela & Jimmy Wu, 2025. "Chronologically Consistent Large Language Models," Papers 2502.21206, arXiv.org, revised Jul 2025.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Firmin Ayivodji & Etienne Briand & Kevin Moran & Dalibor Stevanovic, 2026. "Monetary Policy in the Media Spotlight: Sentiments, Signals, and Economic Impact," Working Papers 26-03, Chair in macroeconomics and forecasting, University of Quebec in Montreal's School of Management.
    2. Alireza Kargarzadeh & Nariman Khaledian & Navid Parvini & Arman Khaledian, 2026. "Large Language Model-Driven Small-Capitalization Trading: Integrating Financial News Sentiment, Macroeconomic Indicators, and Technical Signals," Papers 2608.12283, arXiv.org, revised Sep 2026.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander Eliseev & Sergei Seleznev, 2026. "Fake Date Tests: Can We Trust In-sample Accuracy of LLMs in Macroeconomic Forecasting?," Bank of Russia Working Paper Series wps167, Bank of Russia.
    2. Nikoleta Anesti & Edward Hill & Andreas Joseph, 2026. "Inflation attitudes of large language models," Bank of England Staff Working Paper series 1190, Bank of England.
    3. Nikoleta Anesti & Edward Hill & Andreas Joseph, 2025. "Inflation Attitudes of Large Language Models," Papers 2512.14306, arXiv.org.
    4. Zarifhonarvar, Ali, 2026. "Generating inflation expectations with large language models," Journal of Monetary Economics, Elsevier, vol. 157(C).
    5. Didisheim, Antoine & Fraschini, Martina & Somoza, Luciano, 2025. "AI’s predictable memory in financial analysis," Economics Letters, Elsevier, vol. 256(C).
    6. Hamid Boustanifar & Sasan Mansouri, 2026. "Same Text, Different Numbers: The Divergence of LLM-Based Measures," Papers 2609.31013, arXiv.org.
    7. Leland D. Crane & Akhil Karra & Paul E. Soto, 2025. "Total Recall? Evaluating the Macroeconomic Knowledge of Large Language Models," Finance and Economics Discussion Series 2025-044, Board of Governors of the Federal Reserve System (U.S.).
    8. Dalibor Stevanovic, 2026. "Who Saw It Coming? Historical Experienceand the 2021 Inflation Forecast Failure," Working Papers 26-02, Chair in macroeconomics and forecasting, University of Quebec in Montreal's School of Management.
    9. Ke Wu & Baozhong Yang & Zhenkun Ying & Dexin Zhou, 2025. "Anonymization and Information Loss," Papers 2511.15364, arXiv.org, revised Sep 2026.
    10. Alam, M. Jahangir & Boyle, Shane & Li, Huiyu & Sekhposyan, Tatevik, 2026. "ChatMacro: Evaluating Inflation Forecasts of Generative AI," CEPR Discussion Papers 21057, Centre for Economic Policy Research.
    11. Sophia Kazinnik & Erik Brynjolfsson, 2025. "AI and the Fed," NBER Working Papers 33998, National Bureau of Economic Research, Inc.
    12. repec:rim:rimwps:18-20 is not listed on IDEAS
    13. Nikolaos Giannellis & Stephen G. Hall & Georgios P. Kouretas & George S. Tavlas & Yongli Wang, 2026. "Macroeconomic and Financial Forecasting in the Aftermath of the Covid Shock: Inflation Forecasting With Large Language Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 45(5), pages 2115-2121, August.
    14. John R. Graham & Campbell R. Harvey & Manish Jha, 2026. "CFOs Meet LLMs," Papers 2606.13812, arXiv.org, revised Sep 2026.
    15. Mehmet Caner & Agostino Capponi & Nathan Sun & Jonathan Y. Tan, 2026. "Designing Agentic AI-Based Screening for Portfolio Investment," Papers 2603.23300, arXiv.org, revised Aug 2026.
    16. Zhenyu Gao & Wenxi Jiang & Yutong Yan, 2025. "Detecting Lookahead Bias in LLM Forecasts," Papers 2512.23847, arXiv.org, revised Jun 2026.
    17. Jeffrey D. Michler & Kieran Douglas & Anna Josephson, 2026. "Mining Meaning: Measurement Error in AI-Assisted Literature Reviews," Papers 2609.27686, arXiv.org.
    18. Mostapha Benhenda, 2026. "Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance," Papers 2601.13770, arXiv.org.
    19. Songrun He & Linying Lv & Asaf Manela & Jimmy Wu, 2025. "Instruction Tuning Chronologically Consistent Language Models," Papers 2510.11677, arXiv.org, revised Nov 2025.
    20. Firmin Ayivodji & Etienne Briand & Kevin Moran & Dalibor Stevanovic, 2026. "Monetary Policy in the Media Spotlight: Sentiments, Signals, and Economic Impact," Working Papers 26-03, Chair in macroeconomics and forecasting, University of Quebec in Montreal's School of Management.
    21. Minniti, Antonio & Prettner, Klaus & Venturini, Francesco, 2025. "AI innovation and the labor share in European regions," European Economic Review, Elsevier, vol. 177(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2601.07992. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.