IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v125y2020i2d10.1007_s11192-020-03628-w.html
   My bibliography  Save this article

Can the impact of grey literature be assessed? An investigation of UK government publications cited by articles and books

Author

Listed:
  • Matthew S. Bickley

    (University of Wolverhampton)

  • Kayvan Kousha

    (University of Wolverhampton)

  • Michael Thelwall

    (University of Wolverhampton)

Abstract

Grey literature encompasses a range of relatively informal textual outputs that are not indexed in citation databases. Although they are usually ignored in research evaluations, it is important to develop methods to assess their impact so that their contributions can be recognised, and successful types of grey literature can be encouraged. This article investigates the extent to which 97,150 UK government publications were cited by Scopus articles and Google Books during 2013–2017 in eleven broad subject areas. A method was used to semi-automatically extract citations to the UK government publications from articles and books with high recall and precision. The results showed that Scopus citations are more common than Google Books citations to UK government publications, especially for older documents, and for those in the broad grey literature areas of Healthcare, Education and Science. Since the difference is not huge, both may provide useful grey literature impact data. Precision estimated in two separate ways for both methods appears to show acceptable levels of error for the automated method when compared to a sample with known citation counts.

Suggested Citation

  • Matthew S. Bickley & Kayvan Kousha & Michael Thelwall, 2020. "Can the impact of grey literature be assessed? An investigation of UK government publications cited by articles and books," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1425-1444, November.
    • Handle: RePEc:spr:scient:v:125:y:2020:i:2:d:10.1007_s11192-020-03628-w
    DOI: 10.1007/s11192-020-03628-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03628-w
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03628-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. David Wilkinson & Pardeep Sud & Mike Thelwall, 2014. "Substance without citation: evaluating the online impact of grey literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 797-806, February.
    2. Kayvan Kousha & Mike Thelwall & Somayeh Rezaie, 2011. "Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(11), pages 2147-2164, November.
    3. Kousha, Kayvan & Thelwall, Mike, 2019. "Can Google Scholar and Mendeley help to assess the scholarly impacts of dissertations?," Journal of Informetrics, Elsevier, vol. 13(2), pages 467-484.
    4. Kayvan Kousha & Mike Thelwall, 2015. "An automatic method for extracting citations from Google Books," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(2), pages 309-320, February.
    5. Thelwall, Mike, 2017. "Three practical field normalised alternative indicator formulae for research evaluation," Journal of Informetrics, Elsevier, vol. 11(1), pages 128-151.
    6. Kayvan Kousha & Mike Thelwall & Somayeh Rezaie, 2011. "Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(11), pages 2147-2164, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matthew S. Bickley & Kayvan Kousha & Michael Thelwall, 2022. "A systematic method for identifying references to academic research in grey literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6913-6933, December.
    2. Haunschild, Robin & Bornmann, Lutz, 2023. "Which papers cited which tweets? An exploratory analysis based on Scopus data," Journal of Informetrics, Elsevier, vol. 17(2).
    3. Kayvan Kousha & Mike Thelwall & Matthew Bickley, 2022. "The high scholarly value of grey literature before and during Covid-19," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3489-3504, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mike Thelwall, 2021. "Measuring Societal Impacts Of Research With Altmetrics? Common Problems And Mistakes," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1302-1314, December.
    2. Qingqing Zhou & Chengzhi Zhang, 2020. "Evaluating wider impacts of books via fine-grained mining on citation literatures," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 1923-1948, December.
    3. Ashraf Maleki, 2022. "OCLC library holdings: assessing availability of academic books in libraries in print and electronic compared to citations and altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 991-1020, February.
    4. Ashraf Maleki, 2022. "Why does library holding format really matter for book impact assessment?: Modelling the relationship between citations and altmetrics with print and electronic holdings," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 1129-1160, February.
    5. Mingkun Wei & Abdolreza Noroozi Chakoli, 2020. "Evaluating the relationship between the academic and social impact of open access books based on citation behaviors and social media attention," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2401-2420, December.
    6. Daniel Torres-Salinas & Nicolás Robinson-García & Álvaro Cabezas-Clavijo & Evaristo Jiménez-Contreras, 2014. "Analyzing the citation characteristics of books: edited books, book series and publisher types in the book citation index," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 2113-2127, March.
    7. Mohammadamin Erfanmanesh & A. Noorhidawati & A. Abrizah, 2019. "What can Bookmetrix tell us about the impact of Springer Nature’s books," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 521-536, October.
    8. Tausch, Arno, 2018. "The Market Power of Global Scientific Publishing Companies in the Age of Globalization. An Analysis Based on the OCLC Worldcat," MPRA Paper 87442, University Library of Munich, Germany.
    9. Siluo Yang & Xin Xing & Fan Qi & Maria Cláudia Cabrini Grácio, 2021. "Comparison of academic book impact from a disciplinary perspective: an analysis of citations and altmetric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1101-1123, February.
    10. Boyack, Kevin W. & Klavans, Richard, 2014. "Including cited non-source items in a large-scale map of science: What difference does it make?," Journal of Informetrics, Elsevier, vol. 8(3), pages 569-580.
    11. Muhammad Omar & Arif Mehmood & Gyu Sang Choi & Han Woo Park, 2017. "Global mapping of artificial intelligence in Google and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1269-1305, December.
    12. Ronald Snijder, 2016. "Revisiting an open access monograph experiment: measuring citations and tweets 5 years later," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1855-1875, December.
    13. Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    14. Zhang, Chengzhi & Zhou, Qingqing, 2020. "Assessing books’ depth and breadth via multi-level mining on tables of contents," Journal of Informetrics, Elsevier, vol. 14(2).
    15. Frederik T. Verleysen & Truyken L. B. Ossenblok, 2017. "Profiles of monograph authors in the social sciences and humanities: an analysis of productivity, career stage, co-authorship, disciplinary affiliation and gender, based on a regional bibliographic da," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1673-1686, June.
    16. Emanuel Kulczycki & Przemysław Korytkowski, 2020. "Researchers publishing monographs are more productive and more local-oriented," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1371-1387, November.
    17. Yongjun Zhu & Erjia Yan & Silvio Peroni & Chao Che, 2020. "Nine million book items and eleven million citations: a study of book-based scholarly communication using OpenCitations," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 1097-1112, February.
    18. Qingqing Zhou & Chengzhi Zhang & Star X. Zhao & Bikun Chen, 2016. "Measuring book impact based on the multi-granularity online review mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1435-1455, June.
    19. Elea Giménez-Toledo & Jorge Mañana-Rodríguez & Tim C. E. Engels & Raf Guns & Emanuel Kulczycki & Michael Ochsner & Janne Pölönen & Gunnar Sivertsen & Alesia A. Zuccala, 2019. "Taking scholarly books into account, part II: a comparison of 19 European countries in evaluation and funding," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 233-251, January.
    20. Kousha, Kayvan & Thelwall, Mike, 2018. "Can Microsoft Academic help to assess the citation impact of academic books?," Journal of Informetrics, Elsevier, vol. 12(3), pages 972-984.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:125:y:2020:i:2:d:10.1007_s11192-020-03628-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.