IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0114876.html
   My bibliography  Save this article

Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science

Author

Listed:
  • Coosje L S Veldkamp
  • Michèle B Nuijten
  • Linda Dominguez-Alvarez
  • Marcel A L M van Assen
  • Jelte M Wicherts

Abstract

Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

Suggested Citation

  • Coosje L S Veldkamp & Michèle B Nuijten & Linda Dominguez-Alvarez & Marcel A L M van Assen & Jelte M Wicherts, 2014. "Statistical Reporting Errors and Collaboration on Statistical Analyses in Psychological Science," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-19, December.
  • Handle: RePEc:plo:pone00:0114876
    DOI: 10.1371/journal.pone.0114876
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0114876
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0114876&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0114876?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Blaise Cronin & Debora Shaw & Kathryn La Barre, 2003. "A cast of thousands: Coauthorship and subauthorship collaboration in the 20th century as manifested in the scholarly journal literature of psychology and philosophy," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 54(9), pages 855-871, July.
    2. Jelte M Wicherts & Marjan Bakker & Dylan Molenaar, 2011. "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.
    3. Jelte M. Wicherts, 2011. "Psychology must learn a lesson from fraud case," Nature, Nature, vol. 480(7375), pages 7-7, December.
    4. Marjan Bakker & Jelte M Wicherts, 2014. "Outlier Removal and the Relation with Reporting Errors and Quality of Psychological Research," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-9, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter Pütz & Stephan B. Bruns, 2021. "The (Non‐)Significance Of Reporting Errors In Economics: Evidence From Three Top Journals," Journal of Economic Surveys, Wiley Blackwell, vol. 35(1), pages 348-373, February.
    2. Denes Szucs & John P A Ioannidis, 2017. "Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature," PLOS Biology, Public Library of Science, vol. 15(3), pages 1-18, March.
    3. Klaas Sijtsma & Coosje Veldkamp & Jelte Wicherts, 2016. "Improving the Conduct and Reporting of Statistical Analysis in Psychology," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 33-38, March.
    4. Matteo Colombo & Georgi Duev & Michèle B Nuijten & Jan Sprenger, 2018. "Statistical reporting inconsistencies in experimental philosophy," PLOS ONE, Public Library of Science, vol. 13(4), pages 1-12, April.
    5. Colombo, Matteo & Duev, Georgi & Nuijten, M.B. & Sprenger, Jan, 2018. "Statistical reporting inconsistencies in experimental philosophy," Other publications TiSEM 075f5696-ae1a-4aae-9e17-c, Tilburg University, School of Economics and Management.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matteo Colombo & Georgi Duev & Michèle B Nuijten & Jan Sprenger, 2018. "Statistical reporting inconsistencies in experimental philosophy," PLOS ONE, Public Library of Science, vol. 13(4), pages 1-12, April.
    2. Peter Pütz & Stephan B. Bruns, 2021. "The (Non‐)Significance Of Reporting Errors In Economics: Evidence From Three Top Journals," Journal of Economic Surveys, Wiley Blackwell, vol. 35(1), pages 348-373, February.
    3. Colombo, Matteo & Duev, Georgi & Nuijten, M.B. & Sprenger, Jan, 2018. "Statistical reporting inconsistencies in experimental philosophy," Other publications TiSEM 075f5696-ae1a-4aae-9e17-c, Tilburg University, School of Economics and Management.
    4. Nadine Desrochers & Adèle Paul‐Hus & Jen Pecoskie, 2017. "Five decades of gratitude: A meta‐synthesis of acknowledgments research," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(12), pages 2821-2833, December.
    5. Bruns, Stephan & Herwartz, Helmut & Ioannidis, John P.A. & Islam, Chris-Gabriel & Raters, Fabian H. C., 2023. "Statistical reporting errors in economics," MetaArXiv mbx62, Center for Open Science.
    6. Irwin D. Waldman & Scott O. Lilienfeld, 2016. "Thinking About Data, Research Methods, and Statistical Analyses: Commentary on Sijtsma’s (2014) “Playing with Data”," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 16-26, March.
    7. Franceschet, Massimo & Costantini, Antonio, 2010. "The effect of scholar collaboration on impact and quality of academic papers," Journal of Informetrics, Elsevier, vol. 4(4), pages 540-553.
    8. Sarah Barakat & Sarah Maguire, 2022. "Accessibility of Psychological Treatments for Bulimia Nervosa: A Review of Efficacy and Engagement in Online Self-Help Treatments," IJERPH, MDPI, vol. 20(1), pages 1-27, December.
    9. Wen Lou & Jiangen He & Lingxin Zhang & Zhijie Zhu & Yongjun Zhu, 2023. "Support behind the scenes: the relationship between acknowledgement, coauthor, and citation in Nobel articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5767-5790, October.
    10. Nicola Grassano & Daniele Rotolo & Joshua Hutton & Frédérique Lang & Michael M. Hopkins, 2017. "Funding Data from Publication Acknowledgments: Coverage, Uses, and Limitations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(4), pages 999-1017, April.
    11. Katarina Zigova, 2017. "Specifying Social Weight Matrices of Researcher Networks: The Case of Academic Economists," Working Paper Series of the Department of Economics, University of Konstanz 2017-10, Department of Economics, University of Konstanz.
    12. Michal Krawczyk & Ernesto Reuben, 2012. "(Un)Available upon Request: Field Experiment on Researchers' Willingness to Share Supplementary Materials," Natural Field Experiments 00689, The Field Experiments Website.
    13. Pfenninger, Stefan & DeCarolis, Joseph & Hirth, Lion & Quoilin, Sylvain & Staffell, Iain, 2017. "The importance of open data and software: Is energy research lagging behind?," Energy Policy, Elsevier, vol. 101(C), pages 211-215.
    14. Katherine W. McCain, 2018. "Beyond Garfield’s Citation Index: an assessment of some issues in building a personal name Acknowledgments Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 605-631, February.
    15. Gowri Gopalakrishna & Gerben ter Riet & Gerko Vink & Ineke Stoop & Jelte M Wicherts & Lex M Bouter, 2022. "Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands," PLOS ONE, Public Library of Science, vol. 17(2), pages 1-16, February.
    16. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    17. Ali Gazni & Cassidy R. Sugimoto & Fereshteh Didegah, 2012. "Mapping world scientific collaboration: Authors, institutions, and countries," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(2), pages 323-335, February.
    18. Chenjia Zhang & Yiping Fang & Xiujuan Chen & Tian Congshan, 2019. "Bibliometric Analysis of Trends in Global Sustainable Livelihood Research," Sustainability, MDPI, vol. 11(4), pages 1-28, February.
    19. David Giofrè & Geoff Cumming & Luca Fresc & Ingrid Boedker & Patrizio Tressoldi, 2017. "The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices," PLOS ONE, Public Library of Science, vol. 12(4), pages 1-15, April.
    20. Mario Paolucci & Francisco Grimaldo, 2014. "Mechanism change in a simulation of peer review: from junk support to elitism," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(3), pages 663-688, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0114876. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.