IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v18y2024i1s1751157723001062.html
   My bibliography  Save this article

Exploring the scientific impact of negative results

Author

Listed:
  • Tian, Dan
  • Hu, Xiao
  • Qian, Yuchen
  • Li, Jiang

Abstract

Negative results are a routine part of the scientific research journey, yet they often receive insufficient attention in scientific publications. In this study, we investigate the scientific impact of negative results by comparing the citations and citation context between negative and positive results. Specifically, we compared 159 negative result papers from three journals: Journal of Negative Results in BioMedicine, PLoS One, and BMC Research Notes, with 1,058 matched positive result papers authored by the same first and corresponding authors. The citation context was categorized according to three dimensions: citation aspect, citation purpose, and citation polarity. The first two were automatically provided by Citation Opinion Retrieval and Analysis (CORA), while citation polarity was manually annotated. Our analysis revealed several key findings. Firstly, negative results received 38.6 % fewer citations than positive results, even after controlling for bibliographic factors. Secondly, negative results were associated with a significantly higher proportion of negative citations when compared to positive results. Lastly, a higher proportion of negative results were negatively cited in the methods section.

Suggested Citation

  • Tian, Dan & Hu, Xiao & Qian, Yuchen & Li, Jiang, 2024. "Exploring the scientific impact of negative results," Journal of Informetrics, Elsevier, vol. 18(1).
    • Handle: RePEc:eee:infome:v:18:y:2024:i:1:s1751157723001062
    DOI: 10.1016/j.joi.2023.101481
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157723001062
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2023.101481?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    2. David Colander & Huei-Chun Su, 2015. "Making sense of economists' positive-normative distinction," Journal of Economic Methodology, Taylor & Francis Journals, vol. 22(2), pages 157-170, June.
    3. Daniele Fanelli, 2010. "Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data," PLOS ONE, Public Library of Science, vol. 5(4), pages 1-7, April.
    4. Tokmachev, Andrey M., 2023. "Hidden scales in statistics of citation indicators," Journal of Informetrics, Elsevier, vol. 17(1).
    5. Christian Gumpenberger & Juan Gorraiz & Martin Wieland & Ivana Roche & Edgar Schiebel & Dominique Besagni & Claire François, 2013. "Exploring the bibliometric and semantic nature of negative results," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 277-297, April.
    6. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    7. Daniele Fanelli, 2012. "Negative results are disappearing from most disciplines and countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 891-904, March.
    8. Maarten C. W. Janssen & Mariya Teteryatnikova, 2015. "On the Positive Role of Negative Political Campaigning," Vienna Economics Papers vie1506, University of Vienna, Department of Economics.
    9. Heng Huang & Donghua Zhu & Xuefeng Wang, 2022. "Evaluating scientific impact of publications: combining citation polarity and purpose," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5257-5281, September.
    10. Ying Huang & Donghua Zhu & Qi Lv & Alan L. Porter & Douglas K. R. Robinson & Xuefeng Wang, 2017. "Early insights on the Emerging Sources Citation Index (ESCI): an overlay map-based bibliometric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 2041-2057, June.
    11. Aaron Gerow & Yuening Hu & Jordan Boyd-Graber & David M. Blei & James A. Evans, 2018. "Measuring discursive influence across scholarship," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(13), pages 3308-3313, March.
    12. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    13. Megan L Head & Luke Holman & Rob Lanfear & Andrew T Kahn & Michael D Jennions, 2015. "The Extent and Consequences of P-Hacking in Science," PLOS Biology, Public Library of Science, vol. 13(3), pages 1-15, March.
    14. Hannah Fraser & Tim Parker & Shinichi Nakagawa & Ashley Barnett & Fiona Fidler, 2018. "Questionable research practices in ecology and evolution," PLOS ONE, Public Library of Science, vol. 13(7), pages 1-16, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.
    2. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    3. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    4. Daniele Fanelli & Rodrigo Costas & Vincent Larivière, 2015. "Misconduct Policies, Academic Culture and Career Stage, Not Gender or Pressures to Publish, Affect Scientific Integrity," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-18, June.
    5. Pierre J C Chuard & Milan Vrtílek & Megan L Head & Michael D Jennions, 2019. "Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting?," PLOS Biology, Public Library of Science, vol. 17(1), pages 1-7, January.
    6. van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2018. "P-uniform," MetaArXiv zqjr9, Center for Open Science.
    7. Hladchenko, Myroslava & Moed, Henk F., 2021. "The effect of publication traditions and requirements in research assessment and funding policies upon the use of national journals in 28 post-socialist countries," Journal of Informetrics, Elsevier, vol. 15(4).
    8. Mark D Lindner & Richard K Nakamura, 2015. "Examining the Predictive Validity of NIH Peer Review Scores," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-12, June.
    9. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    10. Augusteijn, Hilde & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2017. "The Effect of Publication Bias on the Assessment of Heterogeneity," OSF Preprints gv25c, Center for Open Science.
    11. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    12. Sergio Copiello, 2020. "The alleged citation advantage of video abstracts may be a matter of self-citations and self-selection bias. Comment on “The impact of video abstract on citation counts” by Zong et al," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 751-757, January.
    13. Salandra, Rossella & Criscuolo, Paola & Salter, Ammon, 2021. "Directing scientists away from potentially biased publications: the role of systematic reviews in health care," Research Policy, Elsevier, vol. 50(1).
    14. Siobhan C Dongés & Jessica M D’Amico & Jane E Butler & Janet L Taylor, 2017. "The effects of cervical transcutaneous spinal direct current stimulation on motor pathways supplying the upper limb in humans," PLOS ONE, Public Library of Science, vol. 12(2), pages 1-20, February.
    15. Joeri K Tijdink & Anton C M Vergouwen & Yvo M Smulders, 2013. "Publication Pressure and Burn Out among Dutch Medical Professors: A Nationwide Survey," PLOS ONE, Public Library of Science, vol. 8(9), pages 1-6, September.
    16. Piotr Bialowolski & Dorota Weziak-Bialowolska & Eileen McNeely, 2021. "The Role of Financial Fragility and Financial Control for Well-Being," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 155(3), pages 1137-1157, June.
    17. Daniele Fanelli & Vincent Larivière, 2016. "Researchers’ Individual Publication Rate Has Not Increased in a Century," PLOS ONE, Public Library of Science, vol. 11(3), pages 1-12, March.
    18. Robbie C M van Aert & Marcel A L M van Assen, 2017. "Bayesian evaluation of effect size after replicating an original study," PLOS ONE, Public Library of Science, vol. 12(4), pages 1-23, April.
    19. Matteo Migheli & Giovanni B. Ramello, 2014. "Open Access Journals & Academics’ Behaviour," ICER Working Papers 03-2014, ICER - International Centre for Economic Research.
    20. Minchul Lee & Min Song, 2020. "Incorporating citation impact into analysis of research trends," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1191-1224, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:18:y:2024:i:1:s1751157723001062. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.