IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v12y2018i3p679-702.html
   My bibliography  Save this article

Author ranking evaluation at scale

Author

Listed:
  • Dunaiski, Marcel
  • Geldenhuys, Jaco
  • Visser, Willem

Abstract

We evaluate author impact indicators and ranking algorithms on two publication databases using large test data sets of well-established researchers. The test data consists of (1) ACM fellowship and (2) various life-time achievement awards. We also evaluate different approaches of dividing credit of papers among co-authors and analyse the impact of self-citations. Furthermore, we evaluate different graph normalisation approaches for when PageRank is computed on author citation graphs.

Suggested Citation

  • Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2018. "Author ranking evaluation at scale," Journal of Informetrics, Elsevier, vol. 12(3), pages 679-702.
    • Handle: RePEc:eee:infome:v:12:y:2018:i:3:p:679-702
    DOI: 10.1016/j.joi.2018.06.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157717303528
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2018.06.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Loet Leydesdorff & Lutz Bornmann & Rüdiger Mutz & Tobias Opthof, 2011. "Turning the tables on citation analysis one more time: Principles for comparing sets of documents," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(7), pages 1370-1381, July.
    2. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Ranking research institutions by the number of highly-cited articles per scientist," Journal of Informetrics, Elsevier, vol. 9(4), pages 915-923.
    3. T. J. Phelan, 1999. "A compendium of issues for citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 45(1), pages 117-136, May.
    4. Mariani, Manuel Sebastian & Medo, Matúš & Zhang, Yi-Cheng, 2016. "Identification of milestone papers through time-balanced network centrality," Journal of Informetrics, Elsevier, vol. 10(4), pages 1207-1223.
    5. Nykl, Michal & Ježek, Karel & Fiala, Dalibor & Dostal, Martin, 2014. "PageRank variants in the evaluation of citation networks," Journal of Informetrics, Elsevier, vol. 8(3), pages 683-692.
    6. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2018. "How to evaluate rankings of academic entities using test data," Journal of Informetrics, Elsevier, vol. 12(3), pages 631-655.
    7. Chao Gao & Zhen Wang & Xianghua Li & Zili Zhang & Wei Zeng, 2016. "PR-Index: Using the h-Index and PageRank for Determining True Impact," PLOS ONE, Public Library of Science, vol. 11(9), pages 1-13, September.
    8. Nykl, Michal & Campr, Michal & Ježek, Karel, 2015. "Author ranking based on personalized PageRank," Journal of Informetrics, Elsevier, vol. 9(4), pages 777-799.
    9. Chen, P. & Xie, H. & Maslov, S. & Redner, S., 2007. "Finding scientific gems with Google’s PageRank algorithm," Journal of Informetrics, Elsevier, vol. 1(1), pages 8-15.
    10. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    11. Fiala, Dalibor & Tutoky, Gabriel, 2017. "PageRank-based prediction of award-winning researchers and the impact of citations," Journal of Informetrics, Elsevier, vol. 11(4), pages 1044-1068.
    12. Jevin D. West & Michael C. Jensen & Ralph J. Dandrea & Gregory J. Gordon & Carl T. Bergstrom, 2013. "Author-level Eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(4), pages 787-801, April.
    13. Fiala, Dalibor & Šubelj, Lovro & Žitnik, Slavko & Bajec, Marko, 2015. "Do PageRank-based author rankings outperform simple citation counts?," Journal of Informetrics, Elsevier, vol. 9(2), pages 334-348.
    14. N. Assimakis & M. Adam, 2010. "A new author’s productivity index: p-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 415-427, November.
    15. Dunaiski, Marcel & Visser, Willem & Geldenhuys, Jaco, 2016. "Evaluating paper and author ranking algorithms using impact and contribution awards," Journal of Informetrics, Elsevier, vol. 10(2), pages 392-407.
    16. Frank J. Trueba & Héctor Guerrero, 2004. "A robust formula to credit authors for their publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(2), pages 181-204, June.
    17. Dalibor Fiala & François Rousselot & Karel Ježek, 2008. "PageRank for bibliographic networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 135-158, July.
    18. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile-based bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    19. Mike Thelwall, 2016. "Interpreting correlations between citation counts and other indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 337-347, July.
    20. Jevin D. West & Michael C. Jensen & Ralph J. Dandrea & Gregory J. Gordon & Carl T. Bergstrom, 2013. "Author‐level Eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(4), pages 787-801, April.
    21. Dag W. Aksnes, 2003. "A macro study of self-citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 56(2), pages 235-246, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kong, Xiangjie & Mao, Mengyi & Jiang, Huizhen & Yu, Shuo & Wan, Liangtian, 2019. "How does collaboration affect researchers’ positions in co-authorship networks?," Journal of Informetrics, Elsevier, vol. 13(3), pages 887-900.
    2. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "Globalised vs averaged: Bias and ranking performance on the author level," Journal of Informetrics, Elsevier, vol. 13(1), pages 299-313.
    3. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2018. "How to evaluate rankings of academic entities using test data," Journal of Informetrics, Elsevier, vol. 12(3), pages 631-655.
    4. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    5. Wang, Jingjing & Xu, Shuqi & Mariani, Manuel S. & Lü, Linyuan, 2021. "The local structure of citation networks uncovers expert-selected milestone papers," Journal of Informetrics, Elsevier, vol. 15(4).
    6. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    7. Tehmina Amjad & Yusra Rehmat & Ali Daud & Rabeeh Ayaz Abbasi, 2020. "Scientific impact of an author and role of self-citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 915-932, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "Globalised vs averaged: Bias and ranking performance on the author level," Journal of Informetrics, Elsevier, vol. 13(1), pages 299-313.
    2. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2018. "How to evaluate rankings of academic entities using test data," Journal of Informetrics, Elsevier, vol. 12(3), pages 631-655.
    3. Dunaiski, Marcel & Visser, Willem & Geldenhuys, Jaco, 2016. "Evaluating paper and author ranking algorithms using impact and contribution awards," Journal of Informetrics, Elsevier, vol. 10(2), pages 392-407.
    4. Xu, Shuqi & Mariani, Manuel Sebastian & Lü, Linyuan & Medo, Matúš, 2020. "Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data," Journal of Informetrics, Elsevier, vol. 14(1).
    5. Dinesh Pradhan & Partha Sarathi Paul & Umesh Maheswari & Subrata Nandi & Tanmoy Chakraborty, 2017. "$$C^3$$ C 3 -index: a PageRank based multi-faceted metric for authors’ performance measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 253-273, January.
    6. Jun Zhang & Zhaolong Ning & Xiaomei Bai & Xiangjie Kong & Jinmeng Zhou & Feng Xia, 2017. "Exploring time factors in measuring the scientific impact of scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1301-1321, September.
    7. Nykl, Michal & Campr, Michal & Ježek, Karel, 2015. "Author ranking based on personalized PageRank," Journal of Informetrics, Elsevier, vol. 9(4), pages 777-799.
    8. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    9. Yu Zhang & Min Wang & Morteza Saberi & Elizabeth Chang, 2022. "Analysing academic paper ranking algorithms using test data and benchmarks: an investigation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4045-4074, July.
    10. Jun Zhang & Yan Hu & Zhaolong Ning & Amr Tolba & Elsayed Elashkar & Feng Xia, 2018. "AIRank: Author Impact Ranking through Positions in Collaboration Networks," Complexity, Hindawi, vol. 2018, pages 1-16, June.
    11. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    12. Fen Zhao & Yi Zhang & Jianguo Lu & Ofer Shai, 2019. "Measuring academic influence using heterogeneous author-citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1119-1140, March.
    13. Ruijie Wang & Yuhao Zhou & An Zeng, 2023. "Evaluating scientists by citation and disruption of their representative works," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1689-1710, March.
    14. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    15. Fiala, Dalibor & Šubelj, Lovro & Žitnik, Slavko & Bajec, Marko, 2015. "Do PageRank-based author rankings outperform simple citation counts?," Journal of Informetrics, Elsevier, vol. 9(2), pages 334-348.
    16. Dejian Yu & Wanru Wang & Shuai Zhang & Wenyu Zhang & Rongyu Liu, 2017. "A multiple-link, mutually reinforced journal-ranking model to measure the prestige of journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 521-542, April.
    17. Yanan Wang & An Zeng & Ying Fan & Zengru Di, 2019. "Ranking scientific publications considering the aging characteristics of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 155-166, July.
    18. Mark Levene & Trevor Fenner & Judit Bar-Ilan, 2019. "Characterisation of the $$\chi$$χ-index and the rec-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 885-896, August.
    19. Jianlin Zhou & An Zeng & Ying Fan & Zengru Di, 2016. "Ranking scientific publications with similarity-preferential mechanism," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 805-816, February.
    20. Marcel Dunaiski & Gillian J. Greene & Bernd Fischer, 2017. "Exploratory search of academic publication and citation data using interactive tag cloud visualizations," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1539-1571, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:12:y:2018:i:3:p:679-702. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.