IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v9y2015i4p800-808.html
   My bibliography  Save this article

Quality versus quantity in scientific impact

Author

Listed:
  • Kaur, Jasleen
  • Ferrara, Emilio
  • Menczer, Filippo
  • Flammini, Alessandro
  • Radicchi, Filippo

Abstract

Citation metrics are becoming pervasive in the quantitative evaluation of scholars, journals, and institutions. Hiring, promotion, and funding decisions increasingly rely on a variety of impact metrics that cannot disentangle quality from quantity of scientific output, and are biased by factors such as discipline and academic age. Biases affecting the evaluation of single papers are compounded when one aggregates citation-based metrics across an entire publication record. It is not trivial to compare the quality of two scholars that during their careers have published at different rates, in different disciplines, and in different periods of time. Here we evaluate a method based on the generation of a statistical baseline specifically tailored on the academic profile of each researcher. We demonstrate the effectiveness of the approach in decoupling the roles of quantity and quality of publications to explain how a certain level of impact is achieved. The method can be extended to simultaneously suppress any source of bias. As an illustration, we use it to capture the quality of the work of Nobel laureates irrespective of number of publications, academic age, and discipline, even when traditional metrics indicate low impact in absolute terms. The procedure is flexible enough to allow for the evaluation of, and fair comparison among, arbitrary collections of papers – scholar publication records, journals, and institutions; in fact, it extends a similar technique that was previously applied to the ranking of research units and countries in specific disciplines (Crespo, Ortuño-Ortí, & Ruiz-Castillo, 2012). We further apply the methodology to almost a million scholars and over six thousand journals to measure the impact that cannot be explained by the volume of publications alone.

Suggested Citation

  • Kaur, Jasleen & Ferrara, Emilio & Menczer, Filippo & Flammini, Alessandro & Radicchi, Filippo, 2015. "Quality versus quantity in scientific impact," Journal of Informetrics, Elsevier, vol. 9(4), pages 800-808.
  • Handle: RePEc:eee:infome:v:9:y:2015:i:4:p:800-808
    DOI: 10.1016/j.joi.2015.07.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157715200120
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ruiz-Castillo, Javier & Albarrán, Pedro, 2012. "The measurement of scientific excellence around the world," UC3M Working papers. Economics we1208, Universidad Carlos III de Madrid. Departamento de Economía.
    2. repec:spr:scient:v:88:y:2011:i:2:d:10.1007_s11192-011-0407-9 is not listed on IDEAS
    3. Loet Leydesdorff & Lutz Bornmann & Rüdiger Mutz & Tobias Opthof, 2011. "Turning the tables on citation analysis one more time: Principles for comparing sets of documents," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(7), pages 1370-1381, July.
    4. Ruiz-Castillo, Javier & Costas, Rodrigo, 2014. "The skewness of scientific productivity," Journal of Informetrics, Elsevier, vol. 8(4), pages 917-934.
    5. Pedro Albarrán & Juan A. Crespo & Ignacio Ortuño & Javier Ruiz-Castillo, 2011. "The skewness of science in 219 sub-fields and a number of aggregates," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(2), pages 385-397, August.
    6. Davis, Paul & Papanek, Gustav F, 1984. "Faculty Ratings of Major Economics Departments by Citations," American Economic Review, American Economic Association, vol. 74(1), pages 225-230, March.
    7. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S. & van Raan, Anthony F.J., 2011. "Towards a new crown indicator: Some theoretical considerations," Journal of Informetrics, Elsevier, vol. 5(1), pages 37-47.
    8. Kaur, Jasleen & Radicchi, Filippo & Menczer, Filippo, 2013. "Universality of scholarly impact metrics," Journal of Informetrics, Elsevier, vol. 7(4), pages 924-932.
    9. Ruiz-Castillo, Javier & Ortuño, Ignacio & Crespo, Juan A., 2012. "The citation merit of scientific publications," UC3M Working papers. Economics we1136, Universidad Carlos III de Madrid. Departamento de Economía.
    10. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mariani, Manuel Sebastian & Medo, Matúš & Zhang, Yi-Cheng, 2016. "Identification of milestone papers through time-balanced network centrality," Journal of Informetrics, Elsevier, vol. 10(4), pages 1207-1223.
    2. Lina M. Cortés & Andrés Mora-Valencia & Javier Perote, 2016. "The productivity of top researchers: a semi-nonparametric approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 891-915, November.
    3. Amjad, Tehmina & Ding, Ying & Xu, Jian & Zhang, Chenwei & Daud, Ali & Tang, Jie & Song, Min, 2017. "Standing on the shoulders of giants," Journal of Informetrics, Elsevier, vol. 11(1), pages 307-323.
    4. repec:spr:scient:v:111:y:2017:i:3:d:10.1007_s11192-017-2374-2 is not listed on IDEAS
    5. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:9:y:2015:i:4:p:800-808. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/joi .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.