IDEAS home Printed from https://ideas.repec.org/p/iso/educat/0066.html
   My bibliography  Save this paper

The Use of Bibliometrics to Measure Research Performance in Education Sciences

Author

Listed:
  • Andrea Diem

    (Swiss Coordination Centre for Research in Education (SCCRE), Aarau)

  • Stefan C. Wolter

    (Swiss Coordination Centre for Research in Education (SCCRE), Aarau, and University of Bern, CESifo and IZA)

Abstract

This paper investigates the fitness-for-purpose and soundness of bibliometric parameters for measuring and elucidating the research performance of individual researchers in the field of education sciences in Switzerland. In order to take into account the specificities of publication practices of researchers in education sciences, the analyses are based on two separate databases: Web of Science and Google Scholar. Both databases show a very unequal distribution of the individual research output, and the indicators used to measure research performance (quantity of publications and citation impact) from the two data sources are highly positively correlated. However, individual characteristics of the researchers, such as age, gender and academic position, that serve to explain the great variance in research performance, can only be identified if the Web of Science is used as a benchmark of research performance. The results indicate that Google Scholar is so inclusive that it impedes a meaningful interpretation of the data. However, the Web of Science inclusion policy for journals is also associated with certain shortcomings that put some researchers at an unjustified disadvantage. Therefore, problems currently exist in regard to both citation databases when used to benchmark individual research performance.

Suggested Citation

  • Andrea Diem & Stefan C. Wolter, 2011. "The Use of Bibliometrics to Measure Research Performance in Education Sciences," Economics of Education Working Paper Series 0066, University of Zurich, Department of Business Administration (IBW), revised May 2013.
  • Handle: RePEc:iso:educat:0066
    as

    Download full text from publisher

    File URL: http://repec.business.uzh.ch/RePEc/iso/leadinghouse/0066_lhwpaper.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "Zeitschriftenrankings für die Wirtschaftswissenschaften – Konstruktion eines umfassenden Metaindexes," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 286-305, August.
    2. John Panaretos & Chrisovaladis Malesios, 2009. "Assessing scientific research performance and impact with single indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 635-670, December.
    3. Jan Ours, 2009. "Will You Still Need Me: When I’m 64?," De Economist, Springer, vol. 157(4), pages 441-460, December.
    4. Carayol, Nicolas & Matt, Mireille, 2006. "Individual and collective determinants of academic scientists' productivity," Information Economics and Policy, Elsevier, vol. 18(1), pages 55-72, March.
    5. Karol Paludkiewicz & Klaus Wohlrabe, 2010. "Qualitätsanalyse von Zeitschriften in den Wirtschaftswissenschaften – über Zitationsdatenbanken und Impaktfaktoren im Online-Zeitalter," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 63(21), pages 18-28, November.
    6. Kristof Witte & Nicky Rogge, 2010. "To publish or not to publish? On the aggregation and drivers of research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 657-680, December.
    7. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    8. Klaus Ritzberger, 2008. "Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 267-285, August.
    9. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    10. Levin, Sharon G & Stephan, Paula E, 1991. "Research Productivity over the Life Cycle: Evidence for Academic Scientists," American Economic Review, American Economic Association, vol. 81(1), pages 114-132, March.
    11. Gonzalez-Brambila, Claudia & Veloso, Francisco M., 2007. "The determinants of research output and impact: A study of Mexican researchers," Research Policy, Elsevier, vol. 36(7), pages 1035-1051, September.
    12. Michael Rauber & Heinrich W. Ursprung, 2008. "Life Cycle and Cohort Productivity in Economic Research: The Case of Germany," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 431-456, November.
    13. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yi Zhang & Mingting Kou & Kaihua Chen & Jiancheng Guan & Yuchen Li, 2016. "Modelling the Basic Research Competitiveness Index (BR-CI) with an application to the biomass energy field," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1221-1241, September.
    2. Fernando J. Garrigos-Simon & M. Dolores Botella-Carrubi & Tomas F. Gonzalez-Cruz, 2018. "Social Capital, Human Capital, and Sustainability: A Bibliometric and Visualization Analysis," Sustainability, MDPI, Open Access Journal, vol. 10(12), pages 1-19, December.
    3. Bayissa Badada Badassa & Baiqing Sun & Lixin Qiao, 2020. "Sustainable Transport Infrastructure and Economic Returns: A Bibliometric and Visualization Analysis," Sustainability, MDPI, Open Access Journal, vol. 12(5), pages 1-24, March.
    4. Akhmat, Ghulam & Zaman, Khalid & Shukui, Tan & Javed, Yasir & Khan, Muhammad Mushtaq, 2014. "Relationship between educational indicators and research outcomes in a panel of top twenty nations: Windows of opportunity," Journal of Informetrics, Elsevier, vol. 8(2), pages 349-361.
    5. Amador Durán-Sánchez & José Álvarez-García & María de la Cruz del Río-Rama & Beatriz Rosado-Cebrián, 2019. "Science Mapping of the Knowledge Base on Tourism Innovation," Sustainability, MDPI, Open Access Journal, vol. 11(12), pages 1-17, June.
    6. Alona Zharova & Janine Tellinger-Rice & Wolfgang Karl Härdle, 2018. "How to Measure a Performance of a Collaborative Research Centre," SFB 649 Discussion Papers SFB649DP2018-001, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    7. Lukas D. Filser & Fábio Francisco Silva & Otávio José Oliveira, 2017. "State of research and future research tendencies in lean healthcare: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 799-816, August.
    8. Rørstad, Kristoffer & Aksnes, Dag W., 2015. "Publication rate expressed by age, gender and academic position – A large-scale analysis of Norwegian academic staff," Journal of Informetrics, Elsevier, vol. 9(2), pages 317-333.
    9. Rodríguez-Soler, Rocío & Uribe-Toril, Juan & De Pablo Valenciano, Jaime, 2020. "Worldwide trends in the scientific production on rural depopulation, a bibliometric analysis using bibliometrix R-tool," Land Use Policy, Elsevier, vol. 97(C).
    10. Jesus Palomo & Cristina Figueroa-Domecq & Pilar Laguna, 2017. "Women, peace and security state-of-art: a bibliometric analysis in social sciences based on SCOPUS database," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 123-148, October.
    11. Alona Zharova & Wolfgang K. Härdle & Stefan Lessmann, 2017. "Is Scientific Performance a Function of Funds?," SFB 649 Discussion Papers SFB649DP2017-028, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    12. Miguel A. García-Pérez, 2015. "Online supplemental information: a sizeable black hole for citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1655-1659, February.
    13. Bornmann, Lutz & Williams, Richard, 2017. "Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on ResearcherID data," Journal of Informetrics, Elsevier, vol. 11(3), pages 788-799.
    14. Fernando J. Garrigos-Simon & Yeamduan Narangajavana-Kaosiri & Ismael Lengua-Lengua, 2018. "Tourism and Sustainability: A Bibliometric and Visualization Analysis," Sustainability, MDPI, Open Access Journal, vol. 10(6), pages 1-23, June.
    15. Feng, Feng & Zhang, Leiyong & Du, Yuneng & Wang, Weiguang, 2015. "Visualization and quantitative study in bibliographic databases: A case in the field of university–industry cooperation," Journal of Informetrics, Elsevier, vol. 9(1), pages 118-134.
    16. Zharova, Alona & Tellinger-Rice, Janine & Härdle, Wolfgang Karl, 2018. "How to Measure a Performance of a Collaborative Research Centre," IRTG 1792 Discussion Papers 2018-011, Humboldt University of Berlin, International Research Training Group 1792 "High Dimensional Nonstationary Time Series".

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bräuninger Michael & Haucap Justus & Muck Johannes, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 12(4), pages 339-371, November.
    2. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "What and How Long Does It Take to Get Tenure? The Case of Economics and Business Administration in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9, pages 473-505, November.
    3. Ali Sina Önder & Sascha Schweitzer, 2017. "Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1297-1331, March.
    4. Maaike Verbree & Edwin Horlings & Peter Groenewegen & Inge Weijden & Peter Besselaar, 2015. "Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 25-49, January.
    5. Jhon Alexánder Méndez Sayago & Leonardo Vera Azaf, 2015. "Salaries, incentives and teaching intellectual production in the public university in Colombia," Revista Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 34(60), pages 95-130, June.
    6. Thomas Bolli & Jörg Schläpfer, 2015. "Job mobility, peer effects, and research productivity in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 629-650, September.
    7. Jhon Alexánder Méndez Sayago & Leonardo Vera Azaf, 2015. "Salarios, incentivos y producción intelectual docente en la universidad pública en Colombia," Revista Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 34(60), pages 95-130, December.
    8. Alexander Butz & Klaus Wohlrabe, 2016. "Die Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc: Methodik, Ergebnisse, Kritik und Vergleich," ifo Working Paper Series 212, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    9. Hartwig, Jochen, 2015. "Structural change, aggregate demand and employment dynamics in the OECD, 1970–2010," Structural Change and Economic Dynamics, Elsevier, vol. 34(C), pages 36-45.
    10. Diniz-Filho, José Alexandre F. & Fioravanti, Maria Clorinda S. & Bini, Luis Mauricio & Rangel, Thiago Fernando, 2016. "Drivers of academic performance in a Brazilian university under a government-restructuring program," Journal of Informetrics, Elsevier, vol. 10(1), pages 151-161.
    11. Vinod Mishra & Russell Smyth, 2013. "Are more senior academics really more research productive than junior academics? Evidence from Australian law schools," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 411-425, August.
    12. Bräuninger, Michael & Haucap, Justus & Muck, Johannes, 2011. "Was lesen und schätzen Ökonomen im Jahr 2011?," DICE Ordnungspolitische Perspektiven 18, University of Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    13. Matthias Krapf, 2010. "Research evaluation and journal quality weights: Much ado about nothing?," Working Paper Series of the Department of Economics, University of Konstanz 2010-02, Department of Economics, University of Konstanz.
    14. Edwin Horlings & Thomas Gurney, 2013. "Search strategies along the academic lifecycle," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1137-1160, March.
    15. Thomas Zwick & Katharina Frosch, 2017. "Attenuation bias when measuring inventive performance," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 26(3), pages 195-201, April.
    16. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    17. Nicolas Carayol, 2006. "La production de brevets par les chercheurs et enseignants-chercheurs.. Le cas de l'université Louis Pasteur," Economie & Prévision, La Documentation Française, vol. 0(4), pages 117-134.
    18. Rivera-Huerta, René & Dutrénit, Gabriela & Ekboir, Javier Mario & Sampedro, José Luis & Vera-Cruz, Alexandre O., 2011. "Do linkages between farmers and academic researchers influence researcher productivity? The Mexican case," Research Policy, Elsevier, vol. 40(7), pages 932-942, September.
    19. João M. Santos & Hugo Horta, 2015. "The generational gap of science: a dynamic cluster analysis of doctorates in an evolving scientific system," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(1), pages 381-406, July.
    20. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.

    More about this item

    Keywords

    bibliometrics; education sciences; research performance; scientometric methods; science research;
    All these keywords.

    JEL classification:

    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I29 - Health, Education, and Welfare - - Education - - - Other

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iso:educat:0066. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/isuzhch.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sara Brunner (email available below). General contact details of provider: https://edirc.repec.org/data/isuzhch.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.