IDEAS home Printed from https://ideas.repec.org/a/wly/sustdv/v33y2025i2p1848-1860.html
   My bibliography  Save this article

Assessing the influence of research quality on policy citations: Quantitative analysis finds non‐academic factors more likely to influence how papers get cited in SDG policy

Author

Listed:
  • Basil Mahfouz
  • Licia Capra
  • Geoff Mulgan

Abstract

Addressing the United Nations Sustainable Development Goals (SDGs) requires policymakers to utilise high‐quality scholarly research effectively. This study explores the extent to which bibliometric indicators of research quality—field weighted citation index, journal CiteScore, author h‐index and mentions—influence the likelihood of SDG‐related research being cited in policy documents. Analysing metadata from over 16 million SDG‐related articles revealed modest relationships between selected bibliometric indicators and policy citations. This suggests that bibliometric measures of research quality may not adequately explain policy influence, or that these metrics might not capture the essence of research excellence from a policy perspective. Importantly, the study identifies variations in research utilisation across different SDGs and highlights the potential role of non‐academic factors in determining policy impact. Results provide a framework for understanding the extent to which policymakers are leveraging the best available evidence in their efforts to meet the SDGs.

Suggested Citation

  • Basil Mahfouz & Licia Capra & Geoff Mulgan, 2025. "Assessing the influence of research quality on policy citations: Quantitative analysis finds non‐academic factors more likely to influence how papers get cited in SDG policy," Sustainable Development, John Wiley & Sons, Ltd., vol. 33(2), pages 1848-1860, April.
    • Handle: RePEc:wly:sustdv:v:33:y:2025:i:2:p:1848-1860
    DOI: 10.1002/sd.3214
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/sd.3214
    Download Restriction: no
    File URL: https://libkey.io/10.1002/sd.3214?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. In Wook Nah & Dai‐Shin Kang & Dae‐Hee Lee & Yun‐Chul Chung, 2009. "A bibliometric evaluation of research performance in different subject categories," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(6), pages 1138-1143, June.
    2. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    3. Frank Biermann & Thomas Hickmann & Carole-Anne Sénit & Marianne Beisheim & Steven Bernstein & Pamela Chasek & Leonie Grob & Rakhyun E. Kim & Louis J. Kotzé & Måns Nilsson & Andrea Ordóñez Llanos & Chu, 2022. "Scientific evidence on the political impact of the Sustainable Development Goals," Nature Sustainability, Nature, vol. 5(9), pages 795-800, September.
    4. Ulrich Schmoch & Torben Schubert, 2009. "When and how to use bibliometrics as a screening tool for research performance," Science and Public Policy, Oxford University Press, vol. 36(10), pages 753-762, December.
    5. Mike Thelwall, 2021. "Measuring Societal Impacts Of Research With Altmetrics? Common Problems And Mistakes," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1302-1314, December.
    6. Rodrigo Costas & Sarah de Rijcke & Noortje Marres, 2021. "“Heterogeneous couplings”: Operationalizing network perspectives to study science‐society interactions through social media metrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(5), pages 595-610, May.
    7. Yian Yin & Yuxiao Dong & Kuansan Wang & Dashun Wang & Benjamin F. Jones, 2022. "Public use and public funding of science," Nature Human Behaviour, Nature, vol. 6(10), pages 1344-1350, October.
    8. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    9. Mike Thelwall & Kayvan Kousha & Mahshid Abdoli & Emma Stuart & Meiko Makita & Paul Wilson & Jonathan Levitt, 2023. "Do altmetric scores reflect article quality? Evidence from the UK Research Excellence Framework 2021," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(5), pages 582-593, May.
    10. Robin Haunschild & Lutz Bornmann, 2017. "How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1209-1216, March.
    11. Lutz Bornmann & Robin Haunschild, 2018. "Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-12, May.
    12. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    13. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2010. "Citations versus journal impact factor as proxy of quality: could the latter ever be preferable?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 821-833, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Hao & Hou, Jianhua, 2024. "Revalidation of the applicability of Altmetrics indicators in article-level evaluation: An empirical analysis of papers of different types of citation trajectories," Journal of Informetrics, Elsevier, vol. 18(4).
    2. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    3. Yang, Siluo & Zheng, Mengxue & Yu, Yonghao & Wolfram, Dietmar, 2021. "Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities?," Journal of Informetrics, Elsevier, vol. 15(1).
    4. Daniela Filippo & Pablo Sastrón-Toledo, 2023. "Influence of research on open science in the public policy sphere," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1995-2017, March.
    5. Drahomira Herrmannova & Robert M. Patton & Petr Knoth & Christopher G. Stahl, 2018. "Do citations and readership identify seminal publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 239-262, April.
    6. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    7. Shahd Al-Janabi & Lee Wei Lim & Luca Aquili, 2021. "Development of a tool to accurately predict UK REF funding allocation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8049-8062, September.
    8. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    9. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2023. "Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1877-1894, March.
    10. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    11. Ou, Guiyan & Zhao, Kang & Zuo, Renxian & Wu, Jiang, 2024. "Effects of research funding on the academic impact and societal visibility of scientific research," Journal of Informetrics, Elsevier, vol. 18(4).
    12. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    13. Hyejin Park & Han Woo Park, 2018. "Research evaluation of Asian countries using altmetrics: comparing South Korea, Japan, Taiwan, Singapore, and China," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 771-788, November.
    14. John P A Ioannidis & Kevin Boyack & Paul F Wouters, 2016. "Citation Metrics: A Primer on How (Not) to Normalize," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-7, September.
    15. Nicky Agate & Rebecca Kennison & Stacy Konkiel & Christopher P. Long & Jason Rhody & Simone Sacchi & Penelope Weber, 2020. "The transformative power of values-enacted scholarship," Palgrave Communications, Palgrave Macmillan, vol. 7(1), pages 1-12, December.
    16. Solanki Gupta & Vivek Kumar Singh & Sumit Kumar Banshal, 2024. "Altmetric data quality analysis using Benford’s law," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4597-4621, July.
    17. Wang, Zuzheng & Lu, Yongxu & Zhou, Yuanyuan & Ji, Jiaojiao, 2024. "SMIAltmetric: A comprehensive metric for evaluating social media impact of scientific papers on Twitter (X)," Journal of Informetrics, Elsevier, vol. 18(3).
    18. Tang, Li & Yang, Defang & Wang, Mingxing & Guo, Ying, 2024. "The mediating impact of citation scope: Evidence from China's ESI publications," Journal of Informetrics, Elsevier, vol. 18(3).
    19. Rongying Zhao & Xu Wang, 2019. "Evaluation and comparison of influence in international Open Access journals between China and USA," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1091-1110, September.
    20. repec:osf:socarx:aep9v_v1 is not listed on IDEAS
    21. González-Betancor, Sara M. & Dorta-González, Pablo, 2023. "Does society show differential attention to researchers based on gender and field?," Journal of Informetrics, Elsevier, vol. 17(4).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:sustdv:v:33:y:2025:i:2:p:1848-1860. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1099-1719 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.