IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v109y2016i2d10.1007_s11192-016-2085-0.html
   My bibliography  Save this article

Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study

Author

Listed:
  • Christopher Carroll

    (University of Sheffield)

Abstract

The “citation score” remains the most commonly-used measure of academic impact, but is also viewed as practically and conceptually limited. The aim of this case study was to test the feasibility of creating a “citation profile” for a single, frequently-cited methods paper, the author’s own publication on the conceptual framework for implementation fidelity. This was a proof-of-concept study that involved an analysis of the citations of a single publication. This analysis involved identifying all citing publications and recording, not only how many times the key paper was cited within each citing publication, but also within which sections of that publication (e.g. Background, Methods, Results etc.). Level of impact could be categorised as high, moderate or low. The key paper had been cited more than 400 times and had a high impact in 25 % of publications based on citation frequency within publications, i.e. the key paper was cited three or more times; and a low impact in 58 % of citing publications, i.e. the key paper was cited just once. There were 41 “high impact” publications based on location of the citations, of which 35 (85 %) were also categorised as high impact by frequency. These results suggest that it is both possible and straightforward to categorise the level of impact of a key paper based on its “citation profile”, i.e., the frequency with which the paper is cited within citing publications, thus adding depth and value to the citation metric.

Suggested Citation

  • Christopher Carroll, 2016. "Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1329-1340, November.
    • Handle: RePEc:spr:scient:v:109:y:2016:i:2:d:10.1007_s11192-016-2085-0
    DOI: 10.1007/s11192-016-2085-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-016-2085-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-016-2085-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Christopher J. Lortie & Lonnie W. Aarssen & Amber E. Budden & Roosa Leimu, 2013. "Do citations and impact factors relate to the real numbers in publications? A case study of citation rates, impact, and effect sizes in ecology and evolutionary biology," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 675-682, February.
    2. Fairclough, Ruth & Thelwall, Mike, 2015. "National research impact indicators from Mendeley readers," Journal of Informetrics, Elsevier, vol. 9(4), pages 845-859.
    3. Bo-Christer Björk & David Solomon, 2015. "Article processing charges in OA journals: relationship between price and quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(2), pages 373-385, May.
    4. Martin Ravallion & Adam Wagstaff, 2011. "On measuring scholarly influence by citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(1), pages 321-337, July.
    5. Thed N. Van Leeuwen & Henk F. Moed & Robert J. W. Tijssen & Martijn S. Visser & Anthony F. J. Van Raan, 2001. "Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 335-346, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lutz Bornmann & Robin Haunschild, 2017. "Quality and impact considerations in bibliometrics: a reply to Ricker (in press)," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1857-1859, June.
    2. Lutz Bornmann & Adam Y. Ye & Fred Y. Ye, 2017. "Sequence analysis of annually normalized citation counts: an empirical analysis based on the characteristic scores and scales (CSS) method," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1665-1680, December.
    3. Tahamtan, Iman & Bornmann, Lutz, 2018. "Core elements in the process of citing publications: Conceptual overview of the literature," Journal of Informetrics, Elsevier, vol. 12(1), pages 203-216.
    4. Samuel-Soma Ajibade & Abdelhamid Zaidi & Asamh Saleh M. Al Luhayb & Anthonia Oluwatosin Adediran & Liton Chandra Voumik & Fazle Rabbi, 2023. "New Insights into the Emerging Trends Research of Machine and Deep Learning Applications in Energy Storage: A Bibliometric Analysis and Publication Trends," International Journal of Energy Economics and Policy, Econjournals, vol. 13(5), pages 303-314, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Neus Herranz & Javier Ruiz-Castillo, 2013. "The end of the “European Paradox”," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 453-464, April.
    2. Amalia Mas-Bleda & Mike Thelwall, 2016. "Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2007-2030, December.
    3. Andre Bruns & Niels Taubert, 2021. "Investigating the Blind Spot of a Monitoring System for Article Processing Charges," Publications, MDPI, vol. 9(3), pages 1-9, September.
    4. Thelwall, Mike, 2016. "The discretised lognormal and hooked power law distributions for complete citation data: Best options for modelling and regression," Journal of Informetrics, Elsevier, vol. 10(2), pages 336-346.
    5. Goodall, Amanda H., 2009. "Highly cited leaders and the performance of research universities," Research Policy, Elsevier, vol. 38(7), pages 1079-1092, September.
    6. Domingo Docampo & Lawrence Cram, 2019. "Highly cited researchers: a moving target," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1011-1025, March.
    7. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    8. Metwaly Ali Mohamed Eldakar, 2019. "Who reads international Egyptian academic articles? An altmetrics analysis of Mendeley readership categories," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 105-135, October.
    9. Yves Gingras & Mahdi Khelfaoui, 2018. "Assessing the effect of the United States’ “citation advantage” on other countries’ scientific impact as measured in the Web of Science (WoS) database," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 517-532, February.
    10. Schreiber, Michael, 2013. "A case study of the arbitrariness of the h-index and the highly-cited-publications indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 379-387.
    11. Henk F. Moed, 2002. "Measuring China"s research performance using the Science Citation Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(3), pages 281-296, March.
    12. Li, Jiang & Qiao, Lili & Li, Wenyuze & Jin, Yidan, 2014. "Chinese-language articles are not biased in citations: Evidences from Chinese-English bilingual journals in Scopus and Web of Science," Journal of Informetrics, Elsevier, vol. 8(4), pages 912-916.
    13. Maziar Montazerian & Edgar Dutra Zanotto & Hellmut Eckert, 2019. "A new parameter for (normalized) evaluation of H-index: countries as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1065-1078, March.
    14. Murat Perit Çakır & Cengiz Acartürk & Oğuzhan Alaşehir & Canan Çilingir, 2015. "A comparative analysis of global and national university ranking systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 813-848, June.
    15. Yutao Sun & Seamus Grimes, 2016. "The emerging dynamic structure of national innovation studies: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 17-40, January.
    16. Zhigao Liu & Yimei Yin & Weidong Liu & Michael Dunford, 2015. "Visualizing the intellectual structure and evolution of innovation systems research: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 135-158, April.
    17. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    18. Gennaro Guida, 2018. "Italian Economics Departments’ Scientific Research Performance: Comparison between VQR and ASN Methodologies," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(9), pages 182-182, August.
    19. David Anderson & John Tressler, 2018. "The Impact of Citation Timing: A Framework and Examples," Review of Economics and Institutions, Università di Perugia, vol. 9(2).
    20. A. Basu & P. Foland & G. Holdridge & R. D. Shelton, 2018. "China’s rising leadership in science and technology: quantitative and qualitative indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 249-269, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:109:y:2016:i:2:d:10.1007_s11192-016-2085-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.