IDEAS home Printed from https://ideas.repec.org/p/iso/educat/0066.html
   My bibliography  Save this paper

The Use of Bibliometrics to Measure Research Performance in Education Sciences

Author

Listed:
  • Andrea Diem

    (Swiss Coordination Centre for Research in Education (SCCRE), Aarau)

  • Stefan C. Wolter

    (Swiss Coordination Centre for Research in Education (SCCRE), Aarau, and University of Bern, CESifo and IZA)

Abstract

This paper investigates the fitness-for-purpose and soundness of bibliometric parameters for measuring and elucidating the research performance of individual researchers in the field of education sciences in Switzerland. In order to take into account the specificities of publication practices of researchers in education sciences, the analyses are based on two separate databases: Web of Science and Google Scholar. Both databases show a very unequal distribution of the individual research output, and the indicators used to measure research performance (quantity of publications and citation impact) from the two data sources are highly positively correlated. However, individual characteristics of the researchers, such as age, gender and academic position, that serve to explain the great variance in research performance, can only be identified if the Web of Science is used as a benchmark of research performance. The results indicate that Google Scholar is so inclusive that it impedes a meaningful interpretation of the data. However, the Web of Science inclusion policy for journals is also associated with certain shortcomings that put some researchers at an unjustified disadvantage. Therefore, problems currently exist in regard to both citation databases when used to benchmark individual research performance.

Suggested Citation

  • Andrea Diem & Stefan C. Wolter, 2011. "The Use of Bibliometrics to Measure Research Performance in Education Sciences," Economics of Education Working Paper Series 0066, University of Zurich, Department of Business Administration (IBW), revised May 2013.
    • Handle: RePEc:iso:educat:0066
    as

    Download full text from publisher

    File URL: http://repec.business.uzh.ch/RePEc/iso/leadinghouse/0066_lhwpaper.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jan Ours, 2009. "Will You Still Need Me: When I’m 64?," De Economist, Springer, vol. 157(4), pages 441-460, December.
    2. Hanna-Mari Puuska, 2010. "Effects of scholar’s gender and professional position on publishing productivity in different publication types. Analysis of a Finnish university," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 419-437, February.
    3. Levin, Sharon G & Stephan, Paula E, 1991. "Research Productivity over the Life Cycle: Evidence for Academic Scientists," American Economic Review, American Economic Association, vol. 81(1), pages 114-132, March.
    4. Michael Rauber & Heinrich W. Ursprung, 2008. "Life Cycle and Cohort Productivity in Economic Research: The Case of Germany," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 431-456, November.
    5. Éric Archambault & Étienne Vignola-Gagné & Grégoire Côté & Vincent Larivière & Yves Gingrasb, 2006. "Benchmarking scientific output in the social sciences and humanities: The limits of existing databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 329-342, September.
    6. Lutz Bornmann & Rüdiger Mutz & Hans‐Dieter Daniel, 2008. "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 830-837, March.
    7. Gonzalez-Brambila, Claudia & Veloso, Francisco M., 2007. "The determinants of research output and impact: A study of Mexican researchers," Research Policy, Elsevier, vol. 36(7), pages 1035-1051, September.
    8. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "Zeitschriftenrankings für die Wirtschaftswissenschaften – Konstruktion eines umfassenden Metaindexes," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 286-305, August.
    9. John Panaretos & Chrisovaladis Malesios, 2009. "Assessing scientific research performance and impact with single indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 635-670, December.
    10. Michael Rauber & Heinrich W. Ursprung, 2008. "Life Cycle and Cohort Productivity in Economic Research: The Case of Germany," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 431-456, November.
    11. Thed van Leeuwen, 2006. "The application of bibliometric analyses in the evaluation of social science research. Who benefits from it, and why it is still feasible," Scientometrics, Springer;Akadémiai Kiadó, vol. 66(1), pages 133-154, January.
    12. Bronwyn Hall & Jacques Mairesse & Laure Turner, 2007. "Identifying Age, Cohort, And Period Effects In Scientific Research Productivity: Discussion And Illustration Using Simulated And Actual Data On French Physicists," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 16(2), pages 159-177.
    13. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    14. Anthony J. Nederhof & Marc Luwel & Henk F. Moed, 2001. "Assessing the quality of scholarly journals in Linguistics:An alternative to citation-based journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 241-265, April.
    15. Anton J. Nederhof, 2006. "Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review," Scientometrics, Springer;Akadémiai Kiadó, vol. 66(1), pages 81-100, January.
    16. A. J. M. Linmans, 2010. "Why with bibliometrics the Humanities does not need to be the weakest link," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(2), pages 337-354, May.
    17. Carayol, Nicolas & Matt, Mireille, 2006. "Individual and collective determinants of academic scientists' productivity," Information Economics and Policy, Elsevier, vol. 18(1), pages 55-72, March.
    18. Karol Paludkiewicz & Klaus Wohlrabe, 2010. "Quality analysis of journals in economics - quotation databases and impact factors in the online age," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 63(21), pages 18-28, November.
    19. Günter Krampen & Ralf Becker & Ute Wahner & Leo Montada, 2007. "On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(2), pages 191-202, May.
    20. Diana Hicks, 1999. "The difficulty of achieving full coverage of international social science literature and the bibliometric consequences," Scientometrics, Springer;Akadémiai Kiadó, vol. 44(2), pages 193-215, February.
    21. Kristof Witte & Nicky Rogge, 2010. "To publish or not to publish? On the aggregation and drivers of research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 657-680, December.
    22. Roberto Todeschini, 2011. "The j-index: a new bibliometric index and multivariate comparisons between other common indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 621-639, June.
    23. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    24. Andrea Bonaccorsi & Cinzia Daraio, 2003. "Age effects in scientific productivity," Scientometrics, Springer;Akadémiai Kiadó, vol. 58(1), pages 49-90, September.
    25. Jung Cheol Shin & William K. Cummings, 2010. "Multilevel analysis of academic publishing across disciplines: research preference, collaboration, and time on research," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 581-594, November.
    26. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    27. Klaus Ritzberger, 2008. "Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 267-285, August.
    28. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    29. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Bräuninger Michael & Haucap Justus & Muck Johannes, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 12(4), pages 339-371, November.
    3. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    4. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "What and How Long Does It Take to Get Tenure? The Case of Economics and Business Administration in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9, pages 473-505, November.
    5. Vinod Mishra & Russell Smyth, 2013. "Are more senior academics really more research productive than junior academics? Evidence from Australian law schools," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 411-425, August.
    6. Ali Sina Önder & Sascha Schweitzer, 2017. "Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1297-1331, March.
    7. Yann Kossi & Jean-Yves Lesueur & Mareva Sabatier, 2016. "Publish or teach? The role of the scientific environment on academics’ multitasking," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 25(3), pages 487-506.
    8. Bäker, Agnes, 2015. "Non-tenured post-doctoral researchers’ job mobility and research output: An analysis of the role of research discipline, department size, and coauthors," Research Policy, Elsevier, vol. 44(3), pages 634-650.
    9. Thomas Bolli & Jörg Schläpfer, 2015. "Job mobility, peer effects, and research productivity in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 629-650, September.
    10. Libman, A., 2010. "Economics in Germany – from National to Global," Journal of the New Economic Association, New Economic Association, issue 8, pages 155-158.
    11. Julia Muschallik & Kerstin Pull, 2016. "Mentoring in higher education: does it enhance mentees’ research productivity?," Education Economics, Taylor & Francis Journals, vol. 24(2), pages 210-223, April.
    12. Maaike Verbree & Edwin Horlings & Peter Groenewegen & Inge Weijden & Peter Besselaar, 2015. "Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 25-49, January.
    13. Giovanni Anania & Annarosa Caruso, 2013. "Two simple new bibliometric indexes to better evaluate research in disciplines where publications typically receive less citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 617-631, August.
    14. Jhon Alexánder Méndez Sayago & Leonardo Vera Azaf, 2015. "Salaries, incentives and teaching intellectual production in the public university in Colombia," Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 34(60), pages 95-130, June.
    15. Andrea Bonaccorsi & Cinzia Daraio & Stefano Fantoni & Viola Folli & Marco Leonetti & Giancarlo Ruocco, 2017. "Do social sciences and humanities behave like life and hard sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 607-653, July.
    16. William E. Savage & Anthony J. Olejniczak, 2021. "Do senior faculty members produce fewer research publications than their younger colleagues? Evidence from Ph.D. granting institutions in the United States," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4659-4686, June.
    17. Jhon Alexánder Méndez Sayago & Leonardo Vera Azaf, 2015. "Salarios, incentivos y producción intelectual docente en la universidad pública en Colombia," Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 34(60), pages 95-130, December.
    18. Chin-Chang Tsai & Elizabeth A. Corley & Barry Bozeman, 2016. "Collaboration experiences across scientific disciplines and cohorts," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 505-529, August.
    19. Raquel Carrasco & Javier Ruiz-Castillo, 2014. "The Evolution Of The Scientific Productivity Of Highly Productive Economists," Economic Inquiry, Western Economic Association International, vol. 52(1), pages 1-16, January.
    20. Guy Madison & Knut Sundell, 2022. "Numbers of publications and citations for researchers in fields pertinent to the social services: a comparison of peer-reviewed journal publications across six disciplines," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(10), pages 6029-6046, October.

    More about this item

    Keywords

    bibliometrics; education sciences; research performance; scientometric methods; science research;
    All these keywords.

    JEL classification:

    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I29 - Health, Education, and Welfare - - Education - - - Other

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iso:educat:0066. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sara Brunner (email available below). General contact details of provider: https://edirc.repec.org/data/isuzhch.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.