IDEAS home Printed from https://ideas.repec.org/p/han/dpaper/dp-485.html
   My bibliography  Save this paper

Quantifying survey expectations: What's wrong with the probability approach?

Author

Listed:
  • Breitung, Jörg
  • Schmeling, Maik

Abstract

We study a matched sample of individual stock market forecasts consisting of both qualitative and quantitative forecasts. This allows us to test for the quality of forecast quantification methods by comparing quantified qualitative forecasts with actual quantitative forecasts. Focusing mainly on the widely used quantification framework advocated by Carlson and Parkin (1975), the so-called "probability approach", we find that quantified expectations derived from the probability approach display a surprisingly weak correlation with reported quantitative stock return forecasts. We trace the reason for this low correlation to the importance of asymmetric and time-varying thresholds, whereas individual heterogeneity across forecasters seems to play a minor role. Hence, our results suggest that qualitative survey data may not be a very useful device to obtain quantitative forecasts and we suggest ways to remedy this problem when designing qualitative surveys.

Suggested Citation

  • Breitung, Jörg & Schmeling, Maik, 2011. "Quantifying survey expectations: What's wrong with the probability approach?," Hannover Economic Papers (HEP) dp-485, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät.
    • Handle: RePEc:han:dpaper:dp-485
    as

    Download full text from publisher

    File URL: http://diskussionspapiere.wiwi.uni-hannover.de/pdf_bib/dp-485.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Dasgupta, Susmita & Lahiri, Kajal, 1992. "A Comparative Study of Alternative Methods of Quantifying Qualitative Survey Responses Using NAPM Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(4), pages 391-400, October.
    2. Silvia Lui & James Mitchell & Martin Weale, 2011. "Qualitative business surveys: signal or noise?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 327-348, April.
    3. Lui, Silvia & Mitchell, James & Weale, Martin, 2011. "The utility of expectational data: Firm-level evidence using matched qualitative-quantitative UK surveys," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1128-1146, October.
    4. Mitchell, James & Mouratidis, Kostas & Weale, Martin, 2007. "Uncertainty in UK manufacturing: Evidence from qualitative survey data," Economics Letters, Elsevier, vol. 94(2), pages 245-252, February.
    5. Pesaran, M. Hashem & Weale, Martin, 2006. "Survey Expectations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 14, pages 715-776, Elsevier.
    6. James Mitchell & Richard J. Smith & Martin R. Weale, 2002. "Quantification of Qualitative Firm-Level Survey Data," Economic Journal, Royal Economic Society, vol. 112(478), pages 117-135, March.
    7. Breitung Jörg, 2008. "Assessing the Rationality of Survey Expectations: The Probability Approach," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 228(5-6), pages 630-643, October.
    8. Jan Marc Berk, 1999. "Measuring inflation expectations: a survey data approach," Applied Economics, Taylor & Francis Journals, vol. 31(11), pages 1467-1480.
    9. N. Gregory Mankiw & Ricardo Reis & Justin Wolfers, 2004. "Disagreement about Inflation Expectations," NBER Chapters, in: NBER Macroeconomics Annual 2003, Volume 18, pages 209-270, National Bureau of Economic Research, Inc.
    10. Graham Elliott & Allan Timmermann & Ivana Komunjer, 2005. "Estimation and Testing of Forecast Rationality under Flexible Loss," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(4), pages 1107-1125.
    11. Loffler, Gunter, 1999. "Refining the Carlson-Parkin method," Economics Letters, Elsevier, vol. 64(2), pages 167-171, August.
    12. Doepke Joerg & Dovern Jonas & Fritsche Ulrich & Slacalek Jiri, 2008. "The Dynamics of European Inflation Expectations," The B.E. Journal of Macroeconomics, De Gruyter, vol. 8(1), pages 1-23, March.
    13. Menkhoff, Lukas & Rebitzky, Rafael R. & Schröder, Michael, 2009. "Heterogeneity in exchange rate expectations: Evidence on the chartist-fundamentalist approach," Journal of Economic Behavior & Organization, Elsevier, vol. 70(1-2), pages 241-252, May.
    14. Claveria, Oscar & Pons, Ernest & Ramos, Raul, 2007. "Business and consumer expectations and macroeconomic forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 47-69.
    15. Frankel, Jeffrey A & Froot, Kenneth A, 1990. "Chartists, Fundamentalists, and Trading in the Foreign Exchange Market," American Economic Review, American Economic Association, vol. 80(2), pages 181-185, May.
    16. Steffen Henzel & Timo Wollmershäuser, 2005. "An Alternative to the Carlson-Parkin Method for the Quantification of Qualitative Inflation Expectations: Evidence from the Ifo World Economic Survey," ifo Working Paper Series 9, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    17. Anderson, Evan W. & Ghysels, Eric & Juergens, Jennifer L., 2009. "The impact of risk and uncertainty on expected returns," Journal of Financial Economics, Elsevier, vol. 94(2), pages 233-263, November.
    18. Defris, L. V. & Williams, R. A., 1979. "Quantitative versus qualitative measures of price expectations: The evidence from Australian consumer surveys," Economics Letters, Elsevier, vol. 2(2), pages 169-173.
    19. Müller, Christian, 2010. "You CAN Carlson-Parkin," Economics Letters, Elsevier, vol. 108(1), pages 33-35, July.
    20. Pesaran, M Hashem, 1985. "Formation of Inflation Expectations in British Manufacturing Industries," Economic Journal, Royal Economic Society, vol. 95(380), pages 948-975, December.
    21. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    22. Thomas Maag, 2009. "On the accuracy of the probability method for quantifying beliefs about inflation," KOF Working papers 09-230, KOF Swiss Economic Institute, ETH Zurich.
    23. James Mitchell & Richard J. Smith & Martin R. Weale, 2005. "Forecasting Manufacturing Output Growth Using Firm‐Level Survey Data," Manchester School, University of Manchester, vol. 73(4), pages 479-499, July.
    24. James Mitchell & Richard J. Smith & Martin R. Weale, 2005. "Forecasting Manufacturing Output Growth Using Firm‐Level Survey Data," Manchester School, University of Manchester, vol. 73(4), pages 479-499, July.
    25. Michela Nardo, 2003. "The Quantification of Qualitative Survey Data: A Critical Assessment," Journal of Economic Surveys, Wiley Blackwell, vol. 17(5), pages 645-668, December.
    26. Das, J.W.M. & Dominitz, J. & van Soest, A.H.O., 1997. "Comparing Predictions and Outcomes : Theory and Application to Income Changes," Other publications TiSEM 6eef11dd-0ae4-4673-b8c0-2, Tilburg University, School of Economics and Management.
    27. repec:bla:econom:v:55:y:1988:i:219:p:317-31 is not listed on IDEAS
    28. Nolte, Ingmar & Pohlmeier, Winfried, 2007. "Using forecasts of forecasters to forecast," International Journal of Forecasting, Elsevier, vol. 23(1), pages 15-28.
    29. Patton, Andrew J. & Timmermann, Allan, 2010. "Why do forecasters disagree? Lessons from the term structure of cross-sectional dispersion," Journal of Monetary Economics, Elsevier, vol. 57(7), pages 803-820, October.
    30. repec:bla:econom:v:42:y:1975:i:166:p:123-38 is not listed on IDEAS
    31. Batchelor, R. A., 1982. "Expectations, output and inflation : The European experience," European Economic Review, Elsevier, vol. 17(1), pages 1-25.
    32. Mitchell, James, 2002. "The use of non-normal distributions in quantifying qualitative survey data on expectations," Economics Letters, Elsevier, vol. 76(1), pages 101-107, June.
    33. Fishe, Raymond P. H. & Lahiri, Kajal, 1981. "On the estimation of inflationary expectations from qualitative responses," Journal of Econometrics, Elsevier, vol. 16(1), pages 89-102, May.
    34. Dr Martin Weale & Dr. James Mitchell, 2005. "Forecasting manufacturing output growth using firm-level survey data," National Institute of Economic and Social Research (NIESR) Discussion Papers 251, National Institute of Economic and Social Research.
    35. Batchelor, R A, 1986. "Quantitative v. Qualitative Measures of Inflation Expectations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 48(2), pages 99-120, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lahiri, Kajal & Zhao, Yongchen, 2015. "Quantifying survey expectations: A critical review and generalization of the Carlson–Parkin method," International Journal of Forecasting, Elsevier, vol. 31(1), pages 51-62.
    2. Oscar Claveria & Enric Monte & Salvador Torra, 2018. "A Data-Driven Approach to Construct Survey-Based Indicators by Means of Evolutionary Algorithms," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 135(1), pages 1-14, January.
    3. Oscar Claveria & Enric Monte & Salvador Torra, 2017. "Let the data do the talking: Empirical modelling of survey-based expectations by means of genetic programming," IREA Working Papers 201711, University of Barcelona, Research Institute of Applied Economics, revised May 2017.
    4. Oscar Claveria & Enric Monte & Salvador Torra, 2019. "Empirical modelling of survey-based expectations for the design of economic indicators in five European regions," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 46(2), pages 205-227, May.
    5. Oscar Claveria & Enric Monte & Salvador Torra, 2015. "“Self-organizing map analysis of agents' expectations. Different patterns of anticipation of the 2008 financial crisis”," IREA Working Papers 201511, University of Barcelona, Research Institute of Applied Economics, revised Mar 2015.
    6. Hutson, Mark & Joutz, Fred & Stekler, Herman, 2014. "Interpreting and evaluating CESIfo's World Economic Survey directional forecasts," Economic Modelling, Elsevier, vol. 38(C), pages 6-11.
    7. Oscar Claveria & Enric Monte & Salvador Torra, 2017. "A new approach for the quantification of qualitative measures of economic expectations," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(6), pages 2685-2706, November.
    8. Kajal Lahiri & Yongchen Zhao, 2013. "Quantifying Heterogeneous Survey Expectations: The Carlson-Parkin Method Revisited," Discussion Papers 13-08, University at Albany, SUNY, Department of Economics.
    9. Swapnil Virendra Chalwadi & Preeti Tushar Joshi & Nitin Mohanlal Sharma & Chaitanya Gite & Sangita Salve, 2023. "Gender Differences in Inflation Expectations: Recent Evidence from India," Administrative Sciences, MDPI, vol. 13(2), pages 1-14, February.
    10. Oscar Claveria & Enric Monte & Salvador Torra, 2018. "Tracking economic growth by evolving expectations via genetic programming: A two-step approach," Working Papers XREAP2018-4, Xarxa de Referència en Economia Aplicada (XREAP), revised Oct 2018.
    11. Christian Seiler, 2015. "On the robustness of balance statistics with respect to nonresponse," OECD Journal: Journal of Business Cycle Measurement and Analysis, OECD Publishing, Centre for International Research on Economic Tendency Surveys, vol. 2014(2), pages 45-62.
    12. Das, Abhiman & Lahiri, Kajal & Zhao, Yongchen, 2019. "Inflation expectations in India: Learning from household tendency surveys," International Journal of Forecasting, Elsevier, vol. 35(3), pages 980-993.
    13. Oscar Claveria & Enric Monte & Salvador Torra, 2019. "Evolutionary Computation for Macroeconomic Forecasting," Computational Economics, Springer;Society for Computational Economics, vol. 53(2), pages 833-849, February.
    14. Alex Botsis & Kevin Lee, 2022. "Nowcasting Using Firm-Level Survey Data; Tracking UK Output Fluctuations and Recessionary Events," Economic Statistics Centre of Excellence (ESCoE) Technical Reports ESCOE-TR-20, Economic Statistics Centre of Excellence (ESCoE).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Oscar Claveria & Enric Monte & Salvador Torra, 2018. "A Data-Driven Approach to Construct Survey-Based Indicators by Means of Evolutionary Algorithms," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 135(1), pages 1-14, January.
    2. Kajal Lahiri & Yongchen Zhao, 2013. "Quantifying Heterogeneous Survey Expectations: The Carlson-Parkin Method Revisited," Discussion Papers 13-08, University at Albany, SUNY, Department of Economics.
    3. Lahiri, Kajal & Zhao, Yongchen, 2015. "Quantifying survey expectations: A critical review and generalization of the Carlson–Parkin method," International Journal of Forecasting, Elsevier, vol. 31(1), pages 51-62.
    4. Oscar Claveria & Enric Monte & Salvador Torra, 2017. "A new approach for the quantification of qualitative measures of economic expectations," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(6), pages 2685-2706, November.
    5. Oscar Claveria & Enric Monte & Salvador Torra, 2018. "Tracking economic growth by evolving expectations via genetic programming: A two-step approach," Working Papers XREAP2018-4, Xarxa de Referència en Economia Aplicada (XREAP), revised Oct 2018.
    6. Oscar Claveria & Enric Monte & Salvador Torra, 2017. "Let the data do the talking: Empirical modelling of survey-based expectations by means of genetic programming," IREA Working Papers 201711, University of Barcelona, Research Institute of Applied Economics, revised May 2017.
    7. Oscar Claveria & Enric Monte & Salvador Torra, 2019. "Evolutionary Computation for Macroeconomic Forecasting," Computational Economics, Springer;Society for Computational Economics, vol. 53(2), pages 833-849, February.
    8. Oscar Claveria, 2021. "Forecasting with Business and Consumer Survey Data," Forecasting, MDPI, vol. 3(1), pages 1-22, February.
    9. Oscar Claveria & Enric Monte & Salvador Torra, 2019. "Empirical modelling of survey-based expectations for the design of economic indicators in five European regions," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 46(2), pages 205-227, May.
    10. Oscar Claveria & Enric Monte & Salvador Torra, 2015. "“Self-organizing map analysis of agents’ expectations. Different patterns of anticipation of the 2008 financial crisis”," AQR Working Papers 201508, University of Barcelona, Regional Quantitative Analysis Group, revised Mar 2015.
    11. Olivier Biau & Hélène Erkel-Rousse & Nicolas Ferrari, 2006. "Réponses individuelles aux enquêtes de conjoncture et prévision de la production manufacturière," Économie et Statistique, Programme National Persée, vol. 395(1), pages 91-116.
    12. Bovi, Maurizio, 2013. "Are the representative agent’s beliefs based on efficient econometric models?," Journal of Economic Dynamics and Control, Elsevier, vol. 37(3), pages 633-648.
    13. Lui, Silvia & Mitchell, James & Weale, Martin, 2011. "The utility of expectational data: Firm-level evidence using matched qualitative-quantitative UK surveys," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1128-1146, October.
    14. Lui, Silvia & Mitchell, James & Weale, Martin, 2011. "The utility of expectational data: Firm-level evidence using matched qualitative-quantitative UK surveys," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1128-1146, October.
    15. Nolte, Ingmar & Pohlmeier, Winfried, 2007. "Using forecasts of forecasters to forecast," International Journal of Forecasting, Elsevier, vol. 23(1), pages 15-28.
    16. Mokinski, Frieder, 2016. "Using time-stamped survey responses to measure expectations at a daily frequency," International Journal of Forecasting, Elsevier, vol. 32(2), pages 271-282.
    17. Sarah M. Lein & Thomas Maag, 2011. "The Formation Of Inflation Perceptions: Some Empirical Facts For European Countries," Scottish Journal of Political Economy, Scottish Economic Society, vol. 58(2), pages 155-188, May.
    18. Silvia Lui & James Mitchell & Martin Weale, 2011. "Qualitative business surveys: signal or noise?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 327-348, April.
    19. Guizzardi, Andrea & Stacchini, Annalisa, 2015. "Real-time forecasting regional tourism with business sentiment surveys," Tourism Management, Elsevier, vol. 47(C), pages 213-223.
    20. Maurizio Bovi, 2016. "The tale of two expectations," Quality & Quantity: International Journal of Methodology, Springer, vol. 50(6), pages 2677-2705, November.

    More about this item

    Keywords

    Quantification; Stock Market Expectations; Probability Approach; Heterogeneity;
    All these keywords.

    JEL classification:

    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • D84 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Expectations; Speculations
    • G17 - Financial Economics - - General Financial Markets - - - Financial Forecasting and Simulation

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:han:dpaper:dp-485. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Heidrich, Christian (email available below). General contact details of provider: https://edirc.repec.org/data/fwhande.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.