IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v37y2021i1p186-204.html
   My bibliography  Save this article

Ranking professional forecasters by the predictive power of their narratives

Author

Listed:
  • Rybinski, Krzysztof

Abstract

This article presents the first ever ranking of professional forecasters based on the predictive power of the narrative of their regular research reports. The ranking is generated by applying the fully automated four-step procedure – called NLP-ForRank – developed in this article. The four steps are data scraping from the internet; data preparation; application of the natural language processing (NLP) models; and evaluation of the predictive power of the NLP indexes with linear regression, Granger causality, vector autoregression (VAR), and random forest forecasting models. Applying this procedure to five large Polish banks and to many time series shows that including the constructed NLP indexes in the forecasting models lowers the forecast errors, and that the optimal model almost always contains the NLP index. The financial news agencies could consider publishing this type of ranking on a regular basis as it would foster accountability, transparency, and a more competitive environment in the professional forecasting industry.

Suggested Citation

  • Rybinski, Krzysztof, 2021. "Ranking professional forecasters by the predictive power of their narratives," International Journal of Forecasting, Elsevier, vol. 37(1), pages 186-204.
    • Handle: RePEc:eee:intfor:v:37:y:2021:i:1:p:186-204
    DOI: 10.1016/j.ijforecast.2020.04.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207020300601
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2020.04.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Grimmer, Justin & Stewart, Brandon M., 2013. "Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts," Political Analysis, Cambridge University Press, vol. 21(3), pages 267-297, July.
    2. Joshua Zoen Git Hiew & Xin Huang & Hao Mou & Duan Li & Qi Wu & Yabo Xu, 2019. "BERT-based Financial Sentiment Index and LSTM-based Stock Return Predictability," Papers 1906.09024, arXiv.org, revised Jul 2022.
    3. Karol Szafranek & Aleksandra Hałka, 2019. "Determinants of Low Inflation in an Emerging, Small Open Economy through the Lens of Aggregated and Disaggregated Approach," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 55(13), pages 3094-3111, October.
    4. David-Jan Jansen & Jakob de Haan, 2010. "An Assessment of the Consistency of ECB Communication using Wordscores," DNB Working Papers 259, Netherlands Central Bank, Research Department.
    5. Frantisek Brazdik & Michal Franta, 2017. "A BVAR Model for Forecasting of Czech Inflation," Working Papers 2017/7, Czech National Bank.
    6. Sims, Christopher A, 1980. "Macroeconomics and Reality," Econometrica, Econometric Society, vol. 48(1), pages 1-48, January.
    7. Adam Elbourne & Jakob de Haan, 2009. "Modeling Monetary Policy Transmission in Acceding Countries: Vector Autoregression Versus Structural Vector Autoregression," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 45(2), pages 4-20, March.
    8. Ingrid E. Fisher & Margaret R. Garnsey & Mark E. Hughes, 2016. "Natural Language Processing in Accounting, Auditing and Finance: A Synthesis of the Literature with a Roadmap for Future Research," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 23(3), pages 157-214, July.
    9. Mirjana Pejić Bach & Živko Krstić & Sanja Seljan & Lejla Turulja, 2019. "Text Mining for Big Data Analysis in Financial Sector: A Literature Review," Sustainability, MDPI, vol. 11(5), pages 1-27, February.
    10. Tim Loughran & Bill Mcdonald, 2016. "Textual Analysis in Accounting and Finance: A Survey," Journal of Accounting Research, Wiley Blackwell, vol. 54(4), pages 1187-1230, September.
    11. Spyros Makridakis & Evangelos Spiliotis & Vassilios Assimakopoulos, 2018. "Statistical and Machine Learning forecasting methods: Concerns and ways forward," PLOS ONE, Public Library of Science, vol. 13(3), pages 1-26, March.
    12. Yuan Luo & William K. Thompson & Timothy M. Herr & Zexian Zeng & Mark A. Berendsen & Siddhartha R. Jonnalagadda & Matthew B. Carson & Justin Starren, 2017. "Natural Language Processing for EHR-Based Pharmacovigilance: A Structured Review," Drug Safety, Springer, vol. 40(11), pages 1075-1089, November.
    13. Rybinski, Krzysztof, 2020. "The forecasting power of the multi-language narrative of sell-side research: A machine learning evaluation," Finance Research Letters, Elsevier, vol. 34(C).
    14. Granger, C W J, 1969. "Investigating Causal Relations by Econometric Models and Cross-Spectral Methods," Econometrica, Econometric Society, vol. 37(3), pages 424-438, July.
    15. Ulrichs Magdalena, 2018. "Identification of Financial and Macroeconomic Shocks in a Var Model of the Polish Economy. A Stability Analysis," Economics and Business Review, Sciendo, vol. 4(1), pages 29-43, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Foltas, Alexander, 2023. "Quantifying priorities in business cycle reports: Analysis of recurring textual patterns around peaks and troughs," Working Papers 44, German Research Foundation's Priority Programme 1859 "Experience and Expectation. Historical Foundations of Economic Behaviour", Humboldt University Berlin.
    2. Tim Köhler & Jörg Döpke, 2023. "Will the last be the first? Ranking German macroeconomic forecasters based on different criteria," Empirical Economics, Springer, vol. 64(2), pages 797-832, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rybinski, Krzysztof, 2020. "The forecasting power of the multi-language narrative of sell-side research: A machine learning evaluation," Finance Research Letters, Elsevier, vol. 34(C).
    2. Senave, Elseline & Jans, Mieke J. & Srivastava, Rajendra P., 2023. "The application of text mining in accounting," International Journal of Accounting Information Systems, Elsevier, vol. 50(C).
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Berkin, Anil & Aerts, Walter & Van Caneghem, Tom, 2023. "Feasibility analysis of machine learning for performance-related attributional statements," International Journal of Accounting Information Systems, Elsevier, vol. 48(C).
    5. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    6. Aaryan Gupta & Vinya Dengre & Hamza Abubakar Kheruwala & Manan Shah, 2020. "Comprehensive review of text-mining applications in finance," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 6(1), pages 1-25, December.
    7. Carl Deschamps & Jan Mattijs, 2015. "Anatomy of a performance management system: the elusive path from targets to productivity," Working Papers CEB 15-037, ULB -- Universite Libre de Bruxelles.
    8. M. T. Alguacil & V. Orts, 2003. "Inward Foreign Direct Investment and Imports in Spain," International Economic Journal, Taylor & Francis Journals, vol. 17(3), pages 19-38.
    9. Ramona Dumitriu & Razvan Stefanescu, 2015. "The Relationship Between Romanian Exports And Economic Growth After The Adhesion To European Union," Risk in Contemporary Economy, "Dunarea de Jos" University of Galati, Faculty of Economics and Business Administration, pages 17-26.
    10. Ericsson, Neil R & Hendry, David F & Mizon, Grayham E, 1998. "Exogeneity, Cointegration, and Economic Policy Analysis," Journal of Business & Economic Statistics, American Statistical Association, vol. 16(4), pages 370-387, October.
    11. Tomasz Woźniak, 2016. "Bayesian Vector Autoregressions," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 49(3), pages 365-380, September.
    12. Christophe Chorro & Emmanuelle Jay & Philippe De Peretti & Thibault Soler, 2021. "Frequency causality measures and Vector AutoRegressive (VAR) models: An improved subset selection method suited to parsimonious systems," Documents de travail du Centre d'Economie de la Sorbonne 21013, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    13. B. Faye & E. Le Fur & S. Prat, 2015. "Dynamics of fine wine and asset prices: evidence from short- and long-run co-movements," Applied Economics, Taylor & Francis Journals, vol. 47(29), pages 3059-3077, June.
    14. Pastwa, Anna M. & Shrestha, Prabal & Thewissen, James & Torsin, Wouter, 2021. "Unpacking the black box of ICO white papers: a topic modeling approach," LIDAM Discussion Papers LFIN 2021018, Université catholique de Louvain, Louvain Finance (LFIN).
    15. Mounir Ben Mbarek & Racha Khairallah & Rochdi Feki, 2015. "Causality relationships between renewable energy, nuclear energy and economic growth in France," Environment Systems and Decisions, Springer, vol. 35(1), pages 133-142, March.
    16. Geweke, J. & Joel Horowitz & Pesaran, M.H., 2006. "Econometrics: A Bird’s Eye View," Cambridge Working Papers in Economics 0655, Faculty of Economics, University of Cambridge.
    17. Neil Gandal & Hanna Halaburda, 2014. "Competition in the Cryptocurrency Market," Staff Working Papers 14-33, Bank of Canada.
    18. Thurik, A. Roy & Carree, Martin A. & van Stel, André & Audretsch, David B., 2008. "Does self-employment reduce unemployment?," Journal of Business Venturing, Elsevier, vol. 23(6), pages 673-686, November.
    19. Jonathan B. Hill, 2007. "Efficient tests of long-run causation in trivariate VAR processes with a rolling window study of the money-income relationship," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(4), pages 747-765.
    20. Helmut Lütkepohl, 2013. "Vector autoregressive models," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 6, pages 139-164, Edward Elgar Publishing.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:37:y:2021:i:1:p:186-204. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.