IDEAS home Printed from https://ideas.repec.org/p/ags/ubfred/316369.html
   My bibliography  Save this paper

The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?

Author

Listed:
  • Heckelei, Thomas
  • Huettel, Silke
  • Odening, Martin
  • Rommel, Jens

Abstract

A vivid debate is ongoing in the scientific community about statistical malpractice and the related publication bias. No general consensus exists on the consequences and this is reflected in heterogeneous rules defined by scientific journals on the use and reporting of statistical inference. This paper aims at discussing how the debate is perceived by the agricultural economics community and implications for our roles as researchers, contributors to the scientific publication process, and teachers. We start by summarizing the current state of the p-value debate and the replication crisis, and commonly applied statistical practices in our community. This is followed by motivation, design, results and discussion of a survey on statistical knowledge and practice among the researchers in the agricultural economics community in Austria, Germany and Switzerland. We conclude that beyond short-term measures like changing rules of reporting in publications, a cultural change regarding empirical scientific practices is needed that stretches across all our roles in the scientific process. Acceptance of scientific work should largely be based on the theoretical and methodological rigor and where the perceived relevance arises from the questions asked, the methodology employed, and the data used but not from the results generated. Revised and clear journal guidelines, the creation of resources for teaching and research, and public recognition of good practice are suggested measures to move forward.

Suggested Citation

  • Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
  • Handle: RePEc:ags:ubfred:316369
    DOI: 10.22004/ag.econ.316369
    as

    Download full text from publisher

    File URL: https://ageconsearch.umn.edu/record/316369/files/Dispap_21_2.pdf
    Download Restriction: no

    File URL: https://ageconsearch.umn.edu/record/316369/files/Dispap_21_2_2.pdf
    Download Restriction: no

    File URL: https://libkey.io/10.22004/ag.econ.316369?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. Stephen T. Ziliak & Deirdre N. McCloskey, 2004. "Size Matters: The Standard Error of Regressions in the American Economic Review," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 331-358, August.
    3. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    4. Jens Rommel & Meike Weltin, 2021. "Is There a Cult of Statistical Significance in Agricultural Economics?," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(3), pages 1176-1191, September.
    5. Sander Greenland, 2019. "Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 106-114, March.
    6. Joachim I. Krueger & Patrick R. Heck, 2019. "Putting the P-Value in its Place," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 122-128, March.
    7. L'industria, 2020. "Call for papers," L'industria, Società editrice il Mulino, issue 2, pages 367-370.
    8. L'industria, 2020. "Call for Papers," L'industria, Società editrice il Mulino, issue 4, pages 787-801.
    9. Abhijit Banerjee & Esther Duflo & Amy Finkelstein & Lawrence F. Katz & Benjamin A. Olken & Anja Sautmann, 2020. "In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for RCTs in Economics," NBER Working Papers 26993, National Bureau of Economic Research, Inc.
    10. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    11. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    12. Ronald D. Fricker & Katherine Burke & Xiaoyan Han & William H. Woodall, 2019. "Assessing the Statistical Analyses Used in Basic and Applied Social Psychology After Their p-Value Ban," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 374-384, March.
    13. Paul J. Ferraro & Pallavi Shukla, 2020. "Feature—Is a Replicability Crisis on the Horizon for Environmental and Resource Economics?," Review of Environmental Economics and Policy, University of Chicago Press, vol. 14(2), pages 339-351.
    14. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "A primer on p-value thresholds and α-levels – two different kettles of fish," SocArXiv d46m2, Center for Open Science.
    15. Guido W. Imbens, 2021. "Statistical Significance, p-Values, and the Reporting of Uncertainty," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 157-174, Summer.
    16. L'industria, 2021. "Call for papers," L'industria, Società editrice il Mulino, issue 4, pages 771-786.
    17. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    18. L'industria, 2021. "Call for paper," L'industria, Società editrice il Mulino, issue 3, pages 579-593.
    19. Andrew Gelman & John Carlin, 2017. "Some Natural Solutions to the -Value Communication Problem—and Why They Won’t Work," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 899-901, July.
    20. Norbert Hirschauer & Oliver Mußhoff & Sven Grüner, 2017. "False Discoveries und Fehlinterpretationen wissenschaftlicher Ergebnisse [False Discoveries and Misinterpretations of Scientific Findings — Implications for Science Communication]," Wirtschaftsdienst, Springer;ZBW - Leibniz Information Centre for Economics, vol. 97(3), pages 201-206, March.
    21. David Giofrè & Geoff Cumming & Luca Fresc & Ingrid Boedker & Patrizio Tressoldi, 2017. "The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices," PLOS ONE, Public Library of Science, vol. 12(4), pages 1-15, April.
    22. Marjan Bakker & Coosje L S Veldkamp & Marcel A L M van Assen & Elise A V Crompvoets & How Hwee Ong & Brian A Nosek & Courtney K Soderberg & David Mellor & Jelte M Wicherts, 2020. "Ensuring the quality and specificity of preregistrations," PLOS Biology, Public Library of Science, vol. 18(12), pages 1-18, December.
    23. The Editors, 2021. "Call for Paper," Action Learning: Research and Practice, Taylor & Francis Journals, vol. 18(3), pages 195-197, September.
    24. Campbell R. Harvey, 2017. "Presidential Address: The Scientific Outlook in Financial Economics," Journal of Finance, American Finance Association, vol. 72(4), pages 1399-1440, August.
    25. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    26. Jonathan W. Schooler, 2014. "Metascience could rescue the ‘replication crisis’," Nature, Nature, vol. 515(7525), pages 9-9, November.
    27. Balazs Aczel & Rink Hoekstra & Andrew Gelman & Eric-Jan Wagenmakers & Irene G. Klugkist & Jeffrey N. Rouder & Joachim Vandekerckhove & Michael D. Lee & Richard D. Morey & Wolf Vanpaemel & Zoltan Diene, 2020. "Discussion points for Bayesian inference," Nature Human Behaviour, Nature, vol. 4(6), pages 561-563, June.
    28. L'industria, 2020. "Call for papers," L'industria, Società editrice il Mulino, issue 1, pages 173-187.
    29. Rahwan, Zoe & Yoeli, Erez & Fasolo, Barbara, 2019. "Heterogeneity in banker culture and its influence on dishonesty," LSE Research Online Documents on Economics 102656, London School of Economics and Political Science, LSE Library.
    30. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Frey Ulrich & Theesfeld Insa & Wagner Peter, 2016. "Die Interpretation des p-Wertes – Grundsätzliche Missverständnisse," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(5), pages 557-575, October.
    31. Zoe Rahwan & Erez Yoeli & Barbara Fasolo, 2019. "Heterogeneity in banker culture and its influence on dishonesty," Nature, Nature, vol. 575(7782), pages 345-349, November.
    32. Geweke, John & Koop, Gary & van Dijk, Herman (ed.), 2011. "The Oxford Handbook of Bayesian Econometrics," OUP Catalogue, Oxford University Press, number 9780199559084, November.
    33. Logg, Jennifer M. & Dorison, Charles A., 2021. "Pre-registration: Weighing costs and benefits for researchers," Organizational Behavior and Human Decision Processes, Elsevier, vol. 167(C), pages 18-27.
    34. Stuart Buck, 2021. "Beware performative reproducibility," Nature, Nature, vol. 595(7866), pages 151-151, July.
    35. Casper Albers, 2019. "The problem with unadjusted multiple and sequential statistical testing," Nature Communications, Nature, vol. 10(1), pages 1-4, December.
    36. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2020. "Can p-values be meaningfully interpreted without random sampling?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 71-91.
    37. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    38. L'industria, 2021. "Call for Papers," L'industria, Società editrice il Mulino, issue 2, pages 377-391.
    39. Valentin Amrhein & Sander Greenland & Blake McShane, 2019. "Scientists rise up against statistical significance," Nature, Nature, vol. 567(7748), pages 305-307, March.
    40. L'industria, 2021. "Call for Papers," L'industria, Società editrice il Mulino, issue 1, pages 175-189.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jens Rommel & Meike Weltin, 2021. "Is There a Cult of Statistical Significance in Agricultural Economics?," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(3), pages 1176-1191, September.
    2. Genevieve LeBaron, 2021. "The Role of Supply Chains in the Global Business of Forced Labour," Journal of Supply Chain Management, Institute for Supply Management, vol. 57(2), pages 29-42, April.
    3. Spiekermann, Sarah & Krasnova, Hanna & Hinz, Oliver & Baumann, Annika & Benlian, Alexander & Gimpel, Henner & Heimbach, Irina & Köster, Antonia & Maedche, Alexander & Niehaves, Björn & Risius, Marten , 2022. "Values and Ethics in Information Systems – A State-of-the-Art Analysis and Avenues for Future Research," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 130842, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    4. Sarah A. Janzen & Jeffrey D. Michler, 2021. "Ulysses' pact or Ulysses' raft: Using pre‐analysis plans in experimental and nonexperimental research," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(4), pages 1286-1304, December.
    5. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    6. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    7. Rainer Alt, 2021. "Electronic Markets on digital platforms and AI," Electronic Markets, Springer;IIM University of St. Gallen, vol. 31(2), pages 233-241, June.
    8. Uwe Hassler & Marc‐Oliver Pohle, 2022. "Unlucky Number 13? Manipulating Evidence Subject to Snooping," International Statistical Review, International Statistical Institute, vol. 90(2), pages 397-410, August.
    9. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    10. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    11. Igor Asanov & Christoph Buehren & Panagiota Zacharodimou, 2020. "The power of experiments: How big is your n?," MAGKS Papers on Economics 202032, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    12. Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2022. "(When) Can We Detect $p$-Hacking?," Papers 2205.07950, arXiv.org.
    13. Todd Mitton, 2022. "Methodological Variation in Empirical Corporate Finance [Identifying and treating outliers in finance]," Review of Financial Studies, Society for Financial Studies, vol. 35(2), pages 527-575.
    14. Christensen, Garret & Miguel, Edward & Sturdy, Jennifer, 2017. "Transparency, Reproducibility, and the Credibility of Economics Research," MetaArXiv 9a3rw, Center for Open Science.
    15. Stephan B. Bruns & David I. Stern, 2019. "Lag length selection and p-hacking in Granger causality testing: prevalence and performance of meta-regression models," Empirical Economics, Springer, vol. 56(3), pages 797-830, March.
    16. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    17. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    18. Zhimin Zhou & Xinyue Ye & Xiangyu Ge, 2017. "The Impacts of Technical Progress on Sulfur Dioxide Kuznets Curve in China: A Spatial Panel Data Approach," Sustainability, MDPI, vol. 9(4), pages 1-27, April.
    19. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    20. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.

    More about this item

    Keywords

    Research Methods/ Statistical Methods;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ags:ubfred:316369. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/zefbnde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: AgEcon Search (email available below). General contact details of provider: https://edirc.repec.org/data/zefbnde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.