IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i7d10.1007_s11192-022-04382-x.html
   My bibliography  Save this article

How accurate are news mentions of scholarly output? A content analysis

Author

Listed:
  • Houqiang Yu

    (Sun Yat-Sen University
    Nanjing University of Science and Technology)

  • Xinyun Yu

    (Nanjing University of Science and Technology)

  • Xueting Cao

    (Jiangsu Broadcasting Corporation)

Abstract

News mentions are considered as useful source for measuring the societal impact of scholarly output, meanwhile data quality plays a fundamental role in its research and application. This study is aimed to measure the accuracy of news mentions data in the altmetrics database, in order to inform the reliability and limitation of relevant news altmetrics studies. In total, 5.83 million news mentions records that involve 1.03 million scholarly outputs were extracted from the whole dataset up to December 2019 provided by the Altmetric database. 3000 records were sampled for content analysis using stratified sampling strategy. Results show that: (1) 6 major types and 14 specific error types are identified. (2) Error occurs in 42.5% of the sample records, 27.1% could be attributable to the news platform and 15.4% could be attributable to the Altmetric database. (3) Inaccessibility to the source news article (25.9%), incorrect news link provided by the Altmetric database (6.9%) and inaccurate news mention (7.9%) are found to be the three most common error types. (4) 8.5% of the sample records have errors that would cause miscalculation and undermine the validity of studies based on the data, while 33.8% of the sample records have errors that would influence the reliability and reproductivity. (5) Underlying reasons for the errors are summarized and possible measures to improve the data quality are discussed in an in-depth and systematic way. These results suggest that although the Altmetric database has made great achievements in collecting news altmetrics data, the data collection can be further improved.

Suggested Citation

  • Houqiang Yu & Xinyun Yu & Xueting Cao, 2022. "How accurate are news mentions of scholarly output? A content analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4075-4096, July.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:7:d:10.1007_s11192-022-04382-x
    DOI: 10.1007/s11192-022-04382-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04382-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04382-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Houqiang Yu & Biegzat Murat & Longfei Li & Tingting Xiao, 2021. "How accurate are Twitter and Facebook altmetrics data? A comparative content analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4437-4463, May.
    2. José Luis Ortega, 2018. "Reliability and accuracy of altmetric providers: a comparison among Altmetric.com, PlumX and Crossref Event Data," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2123-2138, September.
    3. Zhichao Fang & Rodrigo Costas, 2020. "Studying the accumulation velocity of altmetric data tracked by Altmetric.com," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1077-1101, May.
    4. Cassidy R. Sugimoto & Sam Work & Vincent Larivière & Stefanie Haustein, 2017. "Scholarly use of social media and altmetrics: A review of the literature," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2037-2062, September.
    5. Jenny Wooldridge & Mike B. King, 2019. "Altmetric scores: An early indicator of research impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 70(3), pages 271-282, March.
    6. José Luis Ortega, 2020. "Blogs and news sources coverage in altmetrics data providers: a comparative analysis by country, language, and subject," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 555-572, January.
    7. Zahedi, Zohreh & Haustein, Stefanie, 2018. "On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications," Journal of Informetrics, Elsevier, vol. 12(1), pages 191-202.
    8. Kuku Joseph Aduku & Mike Thelwall & Kayvan Kousha, 2017. "Do Mendeley reader counts reflect the scholarly impact of conference papers? An investigation of computer science and engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 573-581, July.
    9. Mojisola Erdt & Aarthy Nagarajan & Sei-Ching Joanna Sin & Yin-Leng Theng, 2016. "Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1117-1166, November.
    10. Houqiang Yu & Xueting Cao & Tingting Xiao & Zhenyi Yang, 2020. "How accurate are policy document mentions? A first look at the role of altmetrics database," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1517-1540, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sergio Copiello, 2020. "Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: tentative findings of a study into the papers retracted by PLoS ONE," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2449-2469, December.
    2. Houqiang Yu & Xueting Cao & Tingting Xiao & Zhenyi Yang, 2020. "How accurate are policy document mentions? A first look at the role of altmetrics database," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1517-1540, November.
    3. Zhichao Fang & Rodrigo Costas & Wencan Tian & Xianwen Wang & Paul Wouters, 2020. "An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2519-2549, September.
    4. Yang, Siluo & Zheng, Mengxue & Yu, Yonghao & Wolfram, Dietmar, 2021. "Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities?," Journal of Informetrics, Elsevier, vol. 15(1).
    5. Enrique Orduña-Malea & Cristina I. Font-Julián, 2022. "Are patents linked on Twitter? A case study of Google patents," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6339-6362, November.
    6. António Correia & Hugo Paredes & Benjamim Fonseca, 2018. "Scientometric analysis of scientific publications in CSCW," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 31-89, January.
    7. Yu, Houqiang & Li, Longfei & Cao, Xueting & Chen, Tao, 2022. "Exploring country's preference over news mentions to academic papers," Journal of Informetrics, Elsevier, vol. 16(4).
    8. Sergio Copiello, 2019. "Research Interest: another undisclosed (and redundant) algorithm by ResearchGate," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 351-360, July.
    9. Chieh Liu & Mu-Hsuan Huang, 2022. "Exploring the relationships between altmetric counts and citations of papers in different academic fields based on co-occurrence analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4939-4958, August.
    10. Qianjin Zong & Yafen Xie & Rongchan Tuo & Jingshi Huang & Yang Yang, 2019. "The impact of video abstract on citation counts: evidence from a retrospective cohort study of New Journal of Physics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1715-1727, June.
    11. Lutz Bornmann & Robin Haunschild & Vanash M Patel, 2020. "Are papers addressing certain diseases perceived where these diseases are prevalent? The proposal to use Twitter data as social-spatial sensors," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-22, November.
    12. Ana-Beatriz Hernández-Lara & Maria-Victoria Sánchez-Rebull & Angels Niñerola, 2021. "Six Sigma in Health Literature, What Matters?," IJERPH, MDPI, vol. 18(16), pages 1-13, August.
    13. Houqiang Yu & Biegzat Murat & Longfei Li & Tingting Xiao, 2021. "How accurate are Twitter and Facebook altmetrics data? A comparative content analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4437-4463, May.
    14. Mike Thelwall, 2021. "Measuring Societal Impacts Of Research With Altmetrics? Common Problems And Mistakes," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1302-1314, December.
    15. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    16. Łukasz Wiechetek & Zbigniew Pastuszak, 2022. "Academic social networks metrics: an effective indicator for university performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1381-1401, March.
    17. Cristina López-Duarte & Marta M. Vidal-Suárez & Belén González-Díaz, 2019. "Cross-national distance and international business: an analysis of the most influential recent models," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 173-208, October.
    18. Siluo Yang & Xin Xing & Dietmar Wolfram, 2018. "Difference in the impact of open-access papers published by China and the USA," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1017-1037, May.
    19. Michael Taylor, 2023. "Slow, slow, quick, quick, slow: five altmetric sources observed over a decade show evolving trends, by research age, attention source maturity and open access status," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(4), pages 2175-2200, April.
    20. Ramani, Ravi S. & Aguinis, Herman & Coyle-Shapiro, Jacqueline A.M., 2022. "Defining, measuring, and rewarding scholarly impact: mind the level of analysis," LSE Research Online Documents on Economics 117286, London School of Economics and Political Science, LSE Library.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:7:d:10.1007_s11192-022-04382-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.