IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v53y2002i1d10.1023_a1014831920172.html
   My bibliography  Save this article

Citation analysis of research performer quality

Author

Listed:
  • Ronald N. Kostoff

    (Office of Naval Research)

Abstract

Background: Citation analysis for evaluative purposes typically requires normalization against some control group of similar papers. Selection of this control group is an open question. Objectives: Gain a better understanding of control group requirements for credible normalization. Approach: Performed citation analysis on prior publications of two proposing research units, to help estimate team research quality. Compared citations of each unit"s publications to citations received by thematically and temporally similar papers. Results: Identification of thematically similar papers was very complex and labor intensive, even with relatively few control papers selected. Conclusions: A credible citation analysis for determining performer or team quality should have the following components: – Multiple technical experts to average out individual bias and subjectivity; – A process for comparing performer or team output papers with a normalization base of similar papers; – A process for retrieving a substantial fraction of candidate normalization base papers; Manual evaluation of many candidate normalization base papers to obtain high thematic similarity and statistical representation.

Suggested Citation

  • Ronald N. Kostoff, 2002. "Citation analysis of research performer quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(1), pages 49-71, January.
    • Handle: RePEc:spr:scient:v:53:y:2002:i:1:d:10.1023_a:1014831920172
    DOI: 10.1023/A:1014831920172
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1023/A:1014831920172
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1023/A:1014831920172?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Robert R. Braam & Henk F. Moed & Anthony F. J. van Raan, 1991. "Mapping of science by combined co‐citation and word analysis. I. Structural aspects," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 42(4), pages 233-251, May.
    2. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    3. Albert, M. B. & Avery, D. & Narin, F. & McAllister, P., 1991. "Direct validation of citation counts as indicators of industrially important patents," Research Policy, Elsevier, vol. 20(3), pages 251-259, June.
    4. Francis Narin & Mark P. Carpenter, 1975. "National publication and citation comparisons," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 26(2), pages 80-93, March.
    5. E. Garfield & I. H. Sher, 1963. "New factors in the evaluation of scientific literature through citation indexing," American Documentation, Wiley Blackwell, vol. 14(3), pages 195-201, July.
    6. Michael H. MacRoberts & Barbara R. MacRoberts, 1989. "Problems of citation analysis: A critical review," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 40(5), pages 342-349, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    2. José María Gómez-Sancho & María Jesús Mancebón-Torrubia, 2010. "A new approach to measuring scientific production in JCR journals and its application to Spanish public universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 271-293, October.
    3. Rafael Aleixandre & Juan Carlos Valderrama & José María Desantes & Antonio J. Torregrosa, 2004. "Identification of information sources and citation patterns in the field of reciprocating internal combustion engines," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 321-336, March.
    4. Guan Jiancheng & Wang Junxia, 2004. "Evaluation and interpretation of knowledge production efficiency," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(1), pages 131-155, January.
    5. Jiancheng Guan & Xia Gao, 2008. "Comparison and evaluation of Chinese research performance in the field of bioinformatics," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(2), pages 357-379, May.
    6. Ying He & Jiancheng Guan, 2008. "Contribution of Chinese publications in computer science: A case study on LNCS," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(3), pages 519-534, June.
    7. Colliander, Cristian & Ahlgren, Per, 2011. "The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments," Journal of Informetrics, Elsevier, vol. 5(1), pages 101-113.
    8. Doussoulin, Jean Pierre & Mougenot, Benoit, 2022. "Mapping mining and ecological distribution conflicts in Latin America, a bibliometric analysis," Resources Policy, Elsevier, vol. 77(C).
    9. Christoph Neuhaus & Hans-Dieter Daniel, 2009. "A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(2), pages 219-229, February.
    10. José María Gómez-Sancho & María Jesús Mancebón-Torrubia, 2009. "The evaluation of scientific production: Towards a neutral impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 435-458, November.
    11. Jonathan Adams & Karen Gurney & Louise Jackson, 2008. "Calibrating the zoom — a test of Zitt’s hypothesis," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(1), pages 81-95, April.
    12. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    13. Andrea Bonaccorsi & Tindaro Cicero & Peter Haddawy & Saeed-UL Hassan, 2017. "Explaining the transatlantic gap in research excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 217-241, January.
    14. Sung-Soo Seol & Jung-Min Park, 2008. "Knowledge sources of innovation studies in Korea: A citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(1), pages 3-20, April.
    15. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    2. T. J. Phelan, 1999. "A compendium of issues for citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 45(1), pages 117-136, May.
    3. Jianhua Hou, 2017. "Exploration into the evolution and historical roots of citation analysis by referenced publication year spectroscopy," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1437-1452, March.
    4. Fairclough, Ruth & Thelwall, Mike, 2015. "National research impact indicators from Mendeley readers," Journal of Informetrics, Elsevier, vol. 9(4), pages 845-859.
    5. Ying He & Jiancheng Guan, 2008. "Contribution of Chinese publications in computer science: A case study on LNCS," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(3), pages 519-534, June.
    6. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    7. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    8. Zohreh Zahedi & Rodrigo Costas & Paul Wouters, 2014. "How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1491-1513, November.
    9. Kun Lu & Isola Ajiferuke & Dietmar Wolfram, 2014. "Extending citer analysis to journal impact evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(1), pages 245-260, July.
    10. McAleer, M.J. & Oláh, J. & Popp, J., 2018. "Pros and Cons of the Impact Factor in a Rapidly Changing Digital World," Econometric Institute Research Papers EI2018-11, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    11. Mora, Luca & Deakin, Mark & Reid, Alasdair, 2019. "Combining co-citation clustering and text-based analysis to reveal the main development paths of smart cities," Technological Forecasting and Social Change, Elsevier, vol. 142(C), pages 56-69.
    12. John Panaretos & Chrisovaladis Malesios, 2009. "Assessing scientific research performance and impact with single indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 635-670, December.
    13. Isola Ajiferuke & Dietmar Wolfram, 2010. "Citer analysis as a measure of research impact: library and information science as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 623-638, June.
    14. Mansour Haghighat & Javad Hayatdavoudi, 2021. "How hot are hot papers? The issue of prolificacy and self-citation stacking," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 565-578, January.
    15. Chihmao Hsieh, 2011. "Explicitly searching for useful inventions: dynamic relatedness and the costs of connecting versus synthesizing," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 381-404, February.
    16. Jianhua Hou & Xiucai Yang & Chaomei Chen, 2020. "Measuring researchers’ potential scholarly impact with structural variations: Four types of researchers in information science (1979–2018)," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-26, June.
    17. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    18. Manajit Chakraborty & Maksym Byshkin & Fabio Crestani, 2020. "Patent citation network analysis: A perspective from descriptive statistics and ERGMs," PLOS ONE, Public Library of Science, vol. 15(12), pages 1-28, December.
    19. Rabishankar Giri & Sabuj Kumar Chaudhuri, 2021. "Ranking journals through the lens of active visibility," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2189-2208, March.
    20. Jos� Lobo & Charlotta Mellander & Kevin Stolarick & Deborah Strumsky, 2014. "The Inventive, the Educated and the Creative: How Do They Affect Metropolitan Productivity?," Industry and Innovation, Taylor & Francis Journals, vol. 21(2), pages 155-177, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:53:y:2002:i:1:d:10.1023_a:1014831920172. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.