IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v28y2012i3p695-711.html
   My bibliography  Save this article

The illusion of predictability: How regression statistics mislead experts

Author

Listed:
  • Soyer, Emre
  • Hogarth, Robin M.

Abstract

Does the manner in which results are presented in empirical studies affect perceptions of the predictability of the outcomes? Noting the predominant role of linear regression analysis in empirical economics, we asked 257 academic economists to make probabilistic inferences based on different presentations of the outputs of this statistical tool. The questions concerned the distribution of the dependent variable, conditional on known values of the independent variable. The answers based on the presentation mode that is standard in the literature demonstrated an illusion of predictability; the outcomes were perceived to be more predictable than could be justified by the model. In particular, many respondents failed to take the error term into account. Adding graphs did not improve the inference. Paradoxically, the respondents were more accurate when only graphs were provided (i.e., no regression statistics). The implications of our study suggest, inter alia, the need to reconsider the way in which empirical results are presented, and the possible provision of easy-to-use simulation tools that would enable readers of empirical papers to make accurate inferences.

Suggested Citation

  • Soyer, Emre & Hogarth, Robin M., 2012. "The illusion of predictability: How regression statistics mislead experts," International Journal of Forecasting, Elsevier, vol. 28(3), pages 695-711.
    • Handle: RePEc:eee:intfor:v:28:y:2012:i:3:p:695-711
    DOI: 10.1016/j.ijforecast.2012.02.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207012000258
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2012.02.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. Stephen T. Ziliak & Deirdre N. McCloskey, 2004. "Size Matters: The Standard Error of Regressions in the American Economic Review," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 331-358, August.
    3. Armstrong, J. Scott, 2007. "Significance tests harm progress in forecasting," International Journal of Forecasting, Elsevier, vol. 23(2), pages 321-327.
    4. Michael C. Jensen, 1968. "The Performance Of Mutual Funds In The Period 1945–1964," Journal of Finance, American Finance Association, vol. 23(2), pages 389-416, May.
    5. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    6. Lawrence, Michael & Makridakis, Spyros, 1989. "Factors affecting judgmental forecasts and confidence intervals," Organizational Behavior and Human Decision Processes, Elsevier, vol. 43(2), pages 172-187, April.
    7. Simon, Herbert A, 1978. "Rationality as Process and as Product of Thought," American Economic Review, American Economic Association, vol. 68(2), pages 1-16, May.
    8. David F. Hendry & Bent Nielsen, 2007. "Preface to Econometric Modeling: A Likelihood Approach," Introductory Chapters, in: Econometric Modeling: A Likelihood Approach, Princeton University Press.
    9. Zellner, Arnold, 2004. "To test or not to test and if so, how?: Comments on "size matters"," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 581-586, November.
    10. Carhart, Mark M, 1997. "On Persistence in Mutual Fund Performance," Journal of Finance, American Finance Association, vol. 52(1), pages 57-82, March.
    11. Baltagi, Badi H., 2007. "Worldwide Econometrics Rankings: 1989–2005," Econometric Theory, Cambridge University Press, vol. 23(5), pages 952-1012, October.
    12. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    2. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    3. Fergus Bolger & Gene Rowe, 2015. "The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 5-11, January.
    4. Green, Kesten C. & Armstrong, J. Scott, 2015. "Simple versus complex forecasting: The evidence," Journal of Business Research, Elsevier, vol. 68(8), pages 1678-1685.
    5. Steffen Roth & Jari Kaivo-Oja, 2016. "Is the future a political economy? Functional analysis of three leading foresight and futures studies journals," Post-Print hal-01475083, HAL.
    6. Soyer, Emre & Hogarth, Robin M., 2015. "The golden rule of forecasting: Objections, refinements, and enhancements," Journal of Business Research, Elsevier, vol. 68(8), pages 1702-1704.
    7. Kim, Jae, 2015. "How to Choose the Level of Significance: A Pedagogical Note," MPRA Paper 66373, University Library of Munich, Germany.
    8. Ang, Huat Bin (Andy) & Woodside, Arch G., 2017. "Is Bart Simpson offering sage advice? A case-based general theory of managers' core self-evaluations and job satisfaction," Journal of Business Research, Elsevier, vol. 74(C), pages 11-37.
    9. Ren, Shengce & Tsai, Huei-Ting & Eisingerich, Andreas B., 2016. "Case-based asymmetric modeling of firms with high versus low outcomes in implementing changes in direction," Journal of Business Research, Elsevier, vol. 69(2), pages 500-507.
    10. Green, Kesten C. & Armstrong, J. Scott & Graefe, Andreas, 2015. "Golden rule of forecasting rearticulated: Forecast unto others as you would have them forecast unto you," Journal of Business Research, Elsevier, vol. 68(8), pages 1768-1771.
    11. Katsikopoulos, Konstantinos V. & Şimşek, Özgür & Buckmann, Marcus & Gigerenzer, Gerd, 2022. "Transparent modeling of influenza incidence: Big data or a single data point from psychological theory?," International Journal of Forecasting, Elsevier, vol. 38(2), pages 613-619.
    12. Hogarth, Robin M. & Soyer, Emre, 2015. "Communicating forecasts: The simplicity of simulated experience," Journal of Business Research, Elsevier, vol. 68(8), pages 1800-1809.
    13. Jae H. Kim & In Choi, 2021. "Choosing the Level of Significance: A Decision‐theoretic Approach," Abacus, Accounting Foundation, University of Sydney, vol. 57(1), pages 27-71, March.
    14. Muhammad Ishaq Bhatti & Jae H. Kim, 2020. "Towards a New Paradigm for Statistical Evidence in the Use of p -Value," Econometrics, MDPI, vol. 9(1), pages 1-3, December.
    15. Cai, Huifen & Boateng, Agyenim & Guney, Yilmaz, 2019. "Host country institutions and firm-level R&D influences: An analysis of European Union FDI in China," Research in International Business and Finance, Elsevier, vol. 47(C), pages 311-326.
    16. Arch G. Woodside & Man-Ling Chang & Cheng-Feng Cheng, 2012. "Government Regulations of Business, Corruption, Reforms, and the Economic Growth of Nations," International Journal of Business and Economics, School of Management Development, Feng Chia University, Taichung, Taiwan, vol. 11(2), pages 127-142, December.
    17. Deirdre N. McCloskey & Stephen T. Ziliak, 2012. "Statistical Significance in the New Tom and the Old Tom: A Reply to Thomas Mayer," Econ Journal Watch, Econ Journal Watch, vol. 9(3), pages 298-308, September.
    18. Armstrong, J. Scott, 2011. "Illusions in Regression Analysis," MPRA Paper 81663, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robin Hogarth & Emre Soyer, 2010. "Econometrics and decision making: Effects of presentation mode," Economics Working Papers 1204, Department of Economics and Business, Universitat Pompeu Fabra.
    2. Martin Gold, 2010. "Fiduciary Finance," Books, Edward Elgar Publishing, number 13813.
    3. Christiane Goodfellow & Dirk Schiereck & Steffen Wippler, 2013. "Are behavioural finance equity funds a superior investment? A note on fund performance and market efficiency," Journal of Asset Management, Palgrave Macmillan, vol. 14(2), pages 111-119, April.
    4. Stephen T. Ziliak & Deirdre N. McCloskey, 2013. "We Agree That Statistical Significance Proves Essentially Nothing: A Rejoinder to Thomas Mayer," Econ Journal Watch, Econ Journal Watch, vol. 10(1), pages 97-107, January.
    5. Timothy R. Wojan & Jason P. Brown & Dayton M. Lambert, 2014. "What to Do about the "Cult of Statistical Significance"? A Renewable Fuel Application using the Neyman-Pearson Protocol," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 36(4), pages 674-695.
    6. Prather, Larry J. & Middleton, Karen L., 2006. "Timing and selectivity of mutual fund managers: An empirical test of the behavioral decision-making theory," Journal of Empirical Finance, Elsevier, vol. 13(3), pages 249-273, June.
    7. Bradley Jones, 2015. "Asset Bubbles: Re-thinking Policy for the Age of Asset Management," IMF Working Papers 2015/027, International Monetary Fund.
    8. Rocciolo, Francesco & Gheno, Andrea & Brooks, Chris, 2022. "Explaining abnormal returns in stock markets: An alpha-neutral version of the CAPM," International Review of Financial Analysis, Elsevier, vol. 82(C).
    9. Nektarios A. Michail & Constantinos I. Massouras, 2014. "Back to Basics: Is Statistical Significance all that Matters?," Working Papers 2014-3, Central Bank of Cyprus.
    10. Massimiliano Caporin & Grégory M. Jannin & Francesco Lisi & Bertrand B. Maillet, 2014. "A Survey On The Four Families Of Performance Measures," Journal of Economic Surveys, Wiley Blackwell, vol. 28(5), pages 917-942, December.
    11. Altman, Morris, 2020. "A more scientific approach to applied economics: Reconstructing statistical, analytical significance, and correlation analysis," Economic Analysis and Policy, Elsevier, vol. 66(C), pages 315-324.
    12. Fernando Rubio, 2005. "Eficiencia De Mercado, Administracion De Carteras De Fondos Y Behavioural Finance," Finance 0503028, University Library of Munich, Germany, revised 23 Jul 2005.
    13. Alexander Kempf & Christoph Merkle & Alexandra Niessen†Ruenzi, 2014. "Low Risk and High Return – Affective Attitudes and Stock Market Expectations," European Financial Management, European Financial Management Association, vol. 20(5), pages 995-1030, November.
    14. Perera, H. Niles & Hurley, Jason & Fahimnia, Behnam & Reisi, Mohsen, 2019. "The human factor in supply chain forecasting: A systematic review," European Journal of Operational Research, Elsevier, vol. 274(2), pages 574-600.
    15. Madanoglu, Melih & Lee, Kyuho & Castrogiovanni, Gary J., 2011. "Franchising and firm financial performance among U.S. restaurants," Journal of Retailing, Elsevier, vol. 87(3), pages 406-417.
    16. Philipp Doerrenberg & Jan Schmitz, 2017. "Tax compliance and information provision. A field experiment with small firms," Journal of Behavioral Economics for Policy, Society for the Advancement of Behavioral Economics (SABE), vol. 1(1), pages 47-54, February.
    17. Dorfleitner, Gregor & Fischer, Lukas & Lung, Carina & Willmertinger, Philipp & Stang, Nico & Dietrich, Natalie, 2018. "To follow or not to follow – An empirical analysis of the returns of actors on social trading platforms," The Quarterly Review of Economics and Finance, Elsevier, vol. 70(C), pages 160-171.
    18. Chakravarty, Sugato & Ray, Rina, 2020. "On short-term institutional trading skill, behavioral biases, and liquidity need," Journal of Corporate Finance, Elsevier, vol. 65(C).
    19. Alexandros Kostakis, 2009. "Performance measures and incentives: loading negative coskewness to outperform the CAPM," The European Journal of Finance, Taylor & Francis Journals, vol. 15(5-6), pages 463-486.
    20. Yu, Bin & Shen, Yifan & Jin, Xuejun & Xu, Qi, 2022. "Does prospect theory explain mutual fund performance? Evidence from China," Pacific-Basin Finance Journal, Elsevier, vol. 73(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:28:y:2012:i:3:p:695-711. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.