IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v51y2022i4s0048733322000129.html
   My bibliography  Save this article

How status of research papers affects the way they are read and cited

Author

Listed:
  • Teplitskiy, Misha
  • Duede, Eamon
  • Menietti, Michael
  • Lakhani, Karim R.

Abstract

Although citations are widely used to measure the influence of scientific works, research shows that many citations serve rhetorical functions and reflect little-to-no influence on the citing authors. If highly cited papers disproportionately attract rhetorical citations then their citation counts may reflect rhetorical usefulness more than influence. Alternatively, researchers may perceive highly cited papers to be of higher quality and invest more effort into reading them, leading to disproportionately substantive citations. We test these arguments using data on 17,154 randomly sampled citations collected via surveys from 9,380 corresponding authors in 15 fields. We find that most citations (54%) had little-to-no influence on the citing authors. However, citations to the most highly cited papers were 2–3 times more likely to denote substantial influence. Experimental and correlational data show a key mechanism: displaying low citation counts lowers perceptions of a paper's quality, and papers with poor perceived quality are read more superficially. The results suggest that higher citation counts lead to more meaningful engagement from readers and, consequently, the most highly cited papers influence the research frontier much more than their raw citation counts imply.

Suggested Citation

  • Teplitskiy, Misha & Duede, Eamon & Menietti, Michael & Lakhani, Karim R., 2022. "How status of research papers affects the way they are read and cited," Research Policy, Elsevier, vol. 51(4).
    • Handle: RePEc:eee:respol:v:51:y:2022:i:4:s0048733322000129
    DOI: 10.1016/j.respol.2022.104484
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733322000129
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2022.104484?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Alison Abbott & David Cyranoski & Nicola Jones & Brendan Maher & Quirin Schiermeier & Richard Van Noorden, 2010. "Metrics: Do metrics matter?," Nature, Nature, vol. 465(7300), pages 860-862, June.
    2. Terrence A. Brooks, 1986. "Evidence of complex citer motivations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 37(1), pages 34-36, January.
    3. Seeber, Marco & Cattaneo, Mattia & Meoli, Michele & Malighetti, Paolo, 2019. "Self-citations as strategic response to the use of metrics for career decisions," Research Policy, Elsevier, vol. 48(2), pages 478-491.
    4. Kyle R. Myers & Wei Yang Tham & Yian Yin & Nina Cohodes & Jerry G. Thursby & Marie C. Thursby & Peter Schiffer & Joseph T. Walsh & Karim R. Lakhani & Dashun Wang, 2020. "Unequal effects of the COVID-19 pandemic on scientists," Nature Human Behaviour, Nature, vol. 4(9), pages 880-883, September.
    5. Dag W. Aksnes, 2006. "Citation rates and perceptions of scientific contribution," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(2), pages 169-185, January.
    6. Iman Tahamtan & Lutz Bornmann, 2019. "What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(3), pages 1635-1684, December.
    7. Bikhchandani, Sushil & Hirshleifer, David & Welch, Ivo, 1992. "A Theory of Fads, Fashion, Custom, and Cultural Change in Informational Cascades," Journal of Political Economy, University of Chicago Press, vol. 100(5), pages 992-1026, October.
    8. Hendrik P. van Dalen & K?ne Henkens, 2005. "Signals in science - On the importance of signaling in gaining attention in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 64(2), pages 209-233, August.
    9. Henk F. Moed & Eugene Garfield, 2004. "In basic science the percentage of “authoritative” references decreases as bibliographies become shorter," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(3), pages 295-303, August.
    10. Tove Faber Frandsen & Jeppe Nicolaisen, 2017. "Citation behavior: A large-scale test of the persuasion by name-dropping hypothesis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(5), pages 1278-1284, May.
    11. Nanda, Ramana & Samila, Sampsa & Sorenson, Olav, 2020. "The persistent effect of initial success: Evidence from venture capital," Journal of Financial Economics, Elsevier, vol. 137(1), pages 231-248.
    12. Amir Rubin & Eran Rubin, 2021. "Systematic Bias in the Progress of Research," Journal of Political Economy, University of Chicago Press, vol. 129(9), pages 2666-2719.
    13. Thijs Bol & Mathijs de Vaan & Arnout van de Rijt, 2018. "The Matthew effect in science funding," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(19), pages 4887-4890, May.
    14. Tahamtan, Iman & Bornmann, Lutz, 2018. "Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references?," Journal of Informetrics, Elsevier, vol. 12(3), pages 906-930.
    15. Béatrice Milard, 2014. "The social circles behind scientific references: Relationships between citing and cited authors in chemistry publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(12), pages 2459-2468, December.
    16. Howard D. White, 2004. "Reward, persuasion, and the Sokal Hoax: A study in citation identities," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(1), pages 93-120, May.
    17. Béatrice Milard & Ludovic Tanguy, 2018. "Citations in Scientific Texts: Do Social Relations Matter?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(11), pages 1380-1395, November.
    18. Horbach, Serge & Aagaard, Kaare & Schneider, Jesper W., 2021. "Meta-Research: How problematic citing practices distort science," MetaArXiv aqyhg, Center for Open Science.
    19. Michael H. MacRoberts & Barbara R. MacRoberts, 1989. "Problems of citation analysis: A critical review," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 40(5), pages 342-349, September.
    20. Liv Langfeldt & Ingvild Reymert & Dag W Aksnes, 2021. "The role of metrics in peer assessments [How Incentives Trickle down: Local Use of a National Bibliometric Indicator System]," Research Evaluation, Oxford University Press, vol. 30(1), pages 112-126.
    21. Radicchi, Filippo & Weissman, Alexander & Bollen, Johan, 2017. "Quantifying perceived impact of scientific publications," Journal of Informetrics, Elsevier, vol. 11(3), pages 704-712.
    22. Günter Krampen & Ralf Becker & Ute Wahner & Leo Montada, 2007. "On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(2), pages 191-202, May.
    23. Mengyi Sun & Jainabou Barry Danfa & Misha Teplitskiy, 2021. "Does double-blind peer-review reduce bias? Evidence from a top computer science conference," Papers 2101.02701, arXiv.org.
    24. Aaron Gerow & Yuening Hu & Jordan Boyd-Graber & David M. Blei & James A. Evans, 2018. "Measuring discursive influence across scholarship," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(13), pages 3308-3313, March.
    25. Charles Oppenheim & Susan P. Renn, 1978. "Highly cited old papers and the reasons why they continue to be cited," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 29(5), pages 225-231, September.
    26. Peiling Wang & Dagobert Soergel, 1998. "A cognitive model of document use during a research project. Study I. Document selection," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 49(2), pages 115-133, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Valentina De Simone & Valentina Di Pasquale & Maria Elena Nenni & Salvatore Miranda, 2023. "Sustainable Production Planning and Control in Manufacturing Contexts: A Bibliometric Review," Sustainability, MDPI, vol. 15(18), pages 1-23, September.
    2. Honglin Bao & Misha Teplitskiy, 2024. "A simulation-based analysis of the impact of rhetorical citations in science," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    3. Sam Arts & Nicola Melluso & Reinhilde Veugelers, 2023. "Beyond Citations: Measuring Novel Scientific Ideas and their Impact in Publication Text," Papers 2309.16437, arXiv.org, revised Nov 2023.
    4. Misha Teplitskiy & Soya Park & Neil Thompson & David Karger, 2022. "Intentional and serendipitous diffusion of ideas: Evidence from academic conferences," Papers 2209.01175, arXiv.org, revised Jan 2024.
    5. Claudia Tania Picinin & Bruno Pedroso & Maik Arnold & Renata Vidart Klafke & Guilherme Moreira Caetano Pinto, 2023. "A Review of the Literature about Sustainability in the Work of the Future: An Overview of Industry 4.0 and Human Resources," Sustainability, MDPI, vol. 15(16), pages 1-21, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    2. Honglin Bao & Misha Teplitskiy, 2024. "A simulation-based analysis of the impact of rhetorical citations in science," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    3. Hou, Jianhua & Wang, Dongyi & Li, Jing, 2022. "A new method for measuring the originality of academic articles based on knowledge units in semantic networks," Journal of Informetrics, Elsevier, vol. 16(3).
    4. Tanzila Ahmed & Ben Johnson & Charles Oppenheim & Catherine Peck, 2004. "Highly cited old papers and the reasons why they continue to be cited. Part II., The 1953 Watson and Crick article on the structure of DNA," Scientometrics, Springer;Akadémiai Kiadó, vol. 61(2), pages 147-156, October.
    5. Jiang, Zhuoren & Lin, Tianqianjin & Huang, Cui, 2023. "Deep representation learning of scientific paper reveals its potential scholarly impact," Journal of Informetrics, Elsevier, vol. 17(1).
    6. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    7. Sehrish Iqbal & Saeed-Ul Hassan & Naif Radi Aljohani & Salem Alelyani & Raheel Nawaz & Lutz Bornmann, 2021. "A decade of in-text citation analysis based on natural language processing and machine learning techniques: an overview of empirical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6551-6599, August.
    8. João A G Moreira & Xiao Han T Zeng & Luís A Nunes Amaral, 2015. "The Distribution of the Asymptotic Number of Citations to Sets of Publications by a Researcher or from an Academic Department Are Consistent with a Discrete Lognormal Model," PLOS ONE, Public Library of Science, vol. 10(11), pages 1-17, November.
    9. Ponomariov, Branco & Toivanen, Hannes, 2014. "Knowledge flows and bases in emerging economy innovation systems: Brazilian research 2005–2009," Research Policy, Elsevier, vol. 43(3), pages 588-596.
    10. Jianhua Hou & Xiucai Yang & Yang Zhang, 2023. "The effect of social media knowledge cascade: an analysis of scientific papers diffusion," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5169-5195, September.
    11. Peter Sjögårde & Fereshteh Didegah, 2022. "The association between topic growth and citation impact of research publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1903-1921, April.
    12. Masaki Eto, 2013. "Evaluations of context-based co-citation searching," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 651-673, February.
    13. Meyer, Matthias & Waldkirch, Rüdiger W. & Duscher, Irina & Just, Alexander, 2018. "Drivers of citations: An analysis of publications in “top” accounting journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 24-46.
    14. Constantin Bürgi & Klaus Wohlrabe, 2022. "The influence of Covid-19 on publications in economics: bibliometric evidence from five working paper series," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5175-5189, September.
    15. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    16. Béatrice Milard & Yoann Pitarch, 2023. "Egocentric cocitation networks and scientific papers destinies," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(4), pages 415-433, April.
    17. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    18. Fairclough, Ruth & Thelwall, Mike, 2015. "National research impact indicators from Mendeley readers," Journal of Informetrics, Elsevier, vol. 9(4), pages 845-859.
    19. Dongqing Lyu & Xuanmin Ruan & Juan Xie & Ying Cheng, 2021. "The classification of citing motivations: a meta-synthesis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3243-3264, April.
    20. Kiran Sharma & Parul Khurana, 2021. "Growth and dynamics of Econophysics: a bibliometric and network analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4417-4436, May.

    More about this item

    Keywords

    Metrics; Citations; Influence; Science; Status;
    All these keywords.

    JEL classification:

    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes
    • O34 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Intellectual Property and Intellectual Capital

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:51:y:2022:i:4:s0048733322000129. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.