IDEAS home Printed from https://ideas.repec.org/a/inm/ororsc/v22y2011i4p1105-1120.html
   My bibliography  Save this article

PERSPECTIVE---Researchers Should Make Thoughtful Assessments Instead of Null-Hypothesis Significance Tests

Author

Listed:
  • Andreas Schwab

    (College of Business, Iowa State University, Ames, Iowa 50011)

  • Eric Abrahamson

    (Columbia Business School, Columbia University, New York, New York 10027)

  • William H. Starbuck

    (Lundquist College of Business, University of Oregon, Eugene, Oregon 97403)

  • Fiona Fidler

    (School of Psychological Science, La Trobe University, Victoria, 3086 Australia)

Abstract

Null-hypothesis significance tests (NHSTs) have received much criticism, especially during the last two decades. Yet many behavioral and social scientists are unaware that NHSTs have drawn increasing criticism, so this essay summarizes key criticisms. The essay also recommends alternative ways of assessing research findings. Although these recommendations are not complex, they do involve ways of thinking that many behavioral and social scientists find novel. Instead of making NHSTs, researchers should adapt their research assessments to specific contexts and specific research goals, and then explain their rationales for selecting assessment indicators. Researchers should show the substantive importance of findings by reporting effect sizes and should acknowledge uncertainty by stating confidence intervals. By comparing data with naïve hypotheses rather than with null hypotheses, researchers can challenge themselves to develop better theories. Parsimonious models are easier to understand, and they generalize more reliably. Robust statistical methods tolerate deviations from assumptions about samples.

Suggested Citation

  • Andreas Schwab & Eric Abrahamson & William H. Starbuck & Fiona Fidler, 2011. "PERSPECTIVE---Researchers Should Make Thoughtful Assessments Instead of Null-Hypothesis Significance Tests," Organization Science, INFORMS, vol. 22(4), pages 1105-1120, August.
  • Handle: RePEc:inm:ororsc:v:22:y:2011:i:4:p:1105-1120
    DOI: 10.1287/orsc.1100.0557
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/orsc.1100.0557
    Download Restriction: no

    File URL: https://libkey.io/10.1287/orsc.1100.0557?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Armstrong, J. Scott, 2007. "Significance tests harm progress in forecasting," International Journal of Forecasting, Elsevier, vol. 23(2), pages 321-327.
    2. Soofi, E.S. & Nystrom, P.C. & Yasai-Ardekani, M., 2009. "Executives' perceived environmental uncertainty shortly after 9/11," Computational Statistics & Data Analysis, Elsevier, vol. 53(9), pages 3502-3515, July.
    3. Elliott, J W, 1973. "A Direct Comparison of Short-Run GNP Forecasting Models," The Journal of Business, University of Chicago Press, vol. 46(1), pages 33-60, January.
    4. Hubbard R. & Bayarri M.J., 2003. "Confusion Over Measures of Evidence (ps) Versus Errors (alphas) in Classical Statistical Testing," The American Statistician, American Statistical Association, vol. 57, pages 171-178, August.
    5. Rosenberg, Barr & Houglet, Michel, 1974. "Error Rates in CRSP and Compustat Data Bases and their Implications," Journal of Finance, American Finance Association, vol. 29(4), pages 1303-1310, September.
    6. Edward H. Bowman, 1984. "Content Analysis of Annual Reports for Corporate Strategy and Risk," Interfaces, INFORMS, vol. 14(1), pages 61-71, February.
    7. Fleiss, J.L., 1986. "Significance tests have a role in epidemiologic research: Reactions to A.M. Walker," American Journal of Public Health, American Public Health Association, vol. 76(5), pages 559-560.
    8. Thomas C. Powell, 2003. "Varieties of competitive parity," Strategic Management Journal, Wiley Blackwell, vol. 24(1), pages 61-86, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter Jaskiewicz & Joern H. Block & James G. Combs & Danny Miller, 2017. "The Effects of Founder and Family Ownership on Hired CEOs’ Incentives and Firm Performance," Entrepreneurship Theory and Practice, , vol. 41(1), pages 73-103, January.
    2. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    3. Block, Joern H. & Miller, Danny & Wagner, Dominik, 2014. "Bayesian methods in family business research," Journal of Family Business Strategy, Elsevier, vol. 5(1), pages 97-104.
    4. Christopher Hansen & Joern Block & Matthias Neuenkirch, 2020. "Family Firm Performance Over The Business Cycle: A Meta‐Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 34(3), pages 476-511, July.
    5. Le Breton-Miller, Isabelle & Miller, Danny, 2023. "Contradiction and disaggregation for family firm research," Journal of Family Business Strategy, Elsevier, vol. 14(1).
    6. Stephen Gorard, 2016. "Damaging Real Lives through Obstinacy: Re-Emphasising Why Significance Testing is Wrong," Sociological Research Online, , vol. 21(1), pages 102-115, February.
    7. Arjen Witteloostuijn, 2020. "New-day statistical thinking: A bold proposal for a radical change in practices," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(2), pages 274-278, March.
    8. Gerrit Anton De Waal & Paul Knott, 2019. "Npd Tools, Thoroughness And Performance In Small Firms," International Journal of Innovation Management (ijim), World Scientific Publishing Co. Pte. Ltd., vol. 23(06), pages 1-26, August.
    9. Davidsson, Per, 2015. "Data replication and extension: A commentary," Journal of Business Venturing Insights, Elsevier, vol. 3(C), pages 12-15.
    10. Andreas Schwab, 2018. "Investigating and Communicating the Uncertainty of Effects: The Power of Graphs," Entrepreneurship Theory and Practice, , vol. 42(6), pages 823-834, November.
    11. Martin Bichler & Ulrich Frank & David Avison & Julien Malaurent & Peter Fettke & Dirk Hovorka & Jan Krämer & Daniel Schnurr & Benjamin Müller & Leena Suhl & Bernhard Thalheim, 2016. "Theories in Business and Information Systems Engineering," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 58(4), pages 291-319, August.
    12. van den Oever, Koen, 2017. "Uncharted waters : A behavioral approach to when, why and which organizational changes are adopted," Other publications TiSEM 0136c8c2-ecdd-4f82-8ca7-d, Tilburg University, School of Economics and Management.
    13. Blakeley B. McShane & David Gal, 2016. "Blinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence," Management Science, INFORMS, vol. 62(6), pages 1707-1718, June.
    14. Katrin Zulauf & Ralf Wagner, 2023. "Countering Negotiation Power Asymmetries by Using the Adjusted Winner Algorithm," SN Operations Research Forum, Springer, vol. 4(1), pages 1-20, March.
    15. Jesper W. Schneider, 2015. "Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 411-432, January.
    16. Andreas Schwab, 2015. "Why All Researchers Should Report Effect Sizes and Their Confidence Intervals: Paving the Way for Meta–Analysis and Evidence–Based Management Practices," Entrepreneurship Theory and Practice, , vol. 39(4), pages 719-725, July.
    17. Paul C. Nystrom & Ehsan S. Soofi, 2012. "Rare, outlier and extreme: beyond the Gaussian model and measures," International Journal of Complexity in Leadership and Management, Inderscience Enterprises Ltd, vol. 2(1/2), pages 6-38.
    18. Block, Joern H. & Jaskiewicz, Peter & Miller, Danny, 2011. "Ownership versus management effects on performance in family and founder companies: A Bayesian reconciliation," Journal of Family Business Strategy, Elsevier, vol. 2(4), pages 232-245.
    19. Davidsson, Per & Gruenhagen, Jan Henrik, 2022. "Two types of entrepreneurship process research revisited: Solidifying the evidence and moving forward," Journal of Business Venturing Insights, Elsevier, vol. 18(C).
    20. Thomas C. Powell, 2020. "Can Quantitative Research Solve Social Problems? Pragmatism and the Ethics of Social Research," Journal of Business Ethics, Springer, vol. 167(1), pages 41-48, November.
    21. Jerker Denrell & Christina Fang & Chengwei Liu, 2015. "Perspective—Chance Explanations in the Management Sciences," Organization Science, INFORMS, vol. 26(3), pages 923-940, June.
    22. Hannah Fraser & Tim Parker & Shinichi Nakagawa & Ashley Barnett & Fiona Fidler, 2018. "Questionable research practices in ecology and evolution," PLOS ONE, Public Library of Science, vol. 13(7), pages 1-16, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jesper W. Schneider, 2015. "Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 411-432, January.
    2. Short, Jeremy C. & Palmer, Timothy B., 2003. "Organizational performance referents: An empirical examination of their content and influences," Organizational Behavior and Human Decision Processes, Elsevier, vol. 90(2), pages 209-224, March.
    3. William Starbuck, 2004. "Methodological Challenges Posed by Measures of Performance," Journal of Management & Governance, Springer;Accademia Italiana di Economia Aziendale (AIDEA), vol. 8(4), pages 337-343, October.
    4. Michael Vössing & Niklas Kühl & Matteo Lind & Gerhard Satzger, 2022. "Designing Transparency for Effective Human-AI Collaboration," Information Systems Frontiers, Springer, vol. 24(3), pages 877-895, June.
    5. Carretta, Alessandro & Farina, Vincenzo & Fiordelisi, Franco & Schwizer, Paola, 2006. "Corporate culture and shareholder value in banking industry," MPRA Paper 8304, University Library of Munich, Germany.
    6. Philip Bromiley, 2009. "A Prospect Theory Model of Resource Allocation," Decision Analysis, INFORMS, vol. 6(3), pages 124-138, September.
    7. Litsiou, Konstantia & Polychronakis, Yiannis & Karami, Azhdar & Nikolopoulos, Konstantinos, 2022. "Relative performance of judgmental methods for forecasting the success of megaprojects," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1185-1196.
    8. Minhas Akbar & Ahsan Akbar & Petra Maresova & Minghui Yang & Hafiz Muhammad Arshad, 2020. "Unraveling the Bankruptcy Risk‒Return Paradox across the Corporate Life Cycle," Sustainability, MDPI, vol. 12(9), pages 1-19, April.
    9. Malcolm Smith & Anita Jamil & Yang Chik Johari & Syahrul Ahmar Ahmad, 2006. "The chairman's statement in Malaysian companies: A test of the obfuscation hypothesis," Asian Review of Accounting, Emerald Group Publishing, vol. 14(1), pages 49-65, July.
    10. Juan Manuel García Lara & Beatriz García Osma & Belén Gill de Albornoz Noguer, 2006. "Effects of database choice on international accounting research," Abacus, Accounting Foundation, University of Sydney, vol. 42(3‐4), pages 426-454, September.
    11. Malcolm Wright & J. Scott Armstrong, 2008. "The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?," Interfaces, INFORMS, vol. 38(2), pages 125-139, April.
    12. Syntetos, Aris A. & Kholidasari, Inna & Naim, Mohamed M., 2016. "The effects of integrating management judgement into OUT levels: In or out of context?," European Journal of Operational Research, Elsevier, vol. 249(3), pages 853-863.
    13. Luis R. Gomez‐Mejia & Marianna Makri & Martin Larraza Kintana, 2010. "Diversification Decisions in Family‐Controlled Firms," Journal of Management Studies, Wiley Blackwell, vol. 47(2), pages 223-252, March.
    14. Lun, Y.H. Venus & Browne, Michael & Lai, Kee-hung & Wong, Christina W.Y. & Cheng, T.C.E., 2011. "Examining the influence of firm performance on business risk-taking and the mediation effect of scale of operations in the container terminal industry," Research in Transportation Economics, Elsevier, vol. 32(1), pages 64-70.
    15. Armstrong, J Scott, 1978. "Forecasting with Econometric Methods: Folklore versus Fact," The Journal of Business, University of Chicago Press, vol. 51(4), pages 549-564, October.
    16. Adusei Jumah & Robert M. Kunst, 2016. "Optimizing time-series forecasts for inflation and interest rates using simulation and model averaging," Applied Economics, Taylor & Francis Journals, vol. 48(45), pages 4366-4378, September.
    17. Crone, Sven F. & Hibon, Michèle & Nikolopoulos, Konstantinos, 2011. "Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction," International Journal of Forecasting, Elsevier, vol. 27(3), pages 635-660.
    18. Jan-Erik Antipin & Farid Jimmy Boumediene & Pär Österholm, 2014. "Forecasting Inflation Using Constant Gain Least Squares," Australian Economic Papers, Wiley Blackwell, vol. 53(1-2), pages 2-15, June.
    19. Rahma Chekkar, 2005. "Communication Financiere Et Analyse De Discours. Une Reflexion Methodologique Autour Du Cas Saint-Gobain," Post-Print halshs-00581152, HAL.
    20. Hyunjung Nam & Won Gyun No & Youngsu Lee, 2017. "Are Commercial Financial Databases Reliable? New Evidence from Korea," Sustainability, MDPI, vol. 9(8), pages 1-23, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ororsc:v:22:y:2011:i:4:p:1105-1120. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.