IDEAS home Printed from https://ideas.repec.org/a/sbr/abstra/v64y2012i4p308-330.html
   My bibliography  Save this article

Relative Performance Measurement of Researchers: The Impact of Data Source Selection

Author

Listed:
  • Matthias Meyer
  • Rüdiger W. Waldkirch
  • Michael A. Zaggl

Abstract

We investigate the impact of choosing between four data sources on the relative performance measurement of scholars’ research output. Our results show that rankings can differ considerably and that the observed correlations are often even lower than those reported in previous studies. We show that data source selection generally has a higher effect than does measure selection and that performance measures based on Google Scholar might provide information that complements the more traditional data sources. We find greater differences for accounting, which suggests that decision makers must pay more careful atten-tion when using rankings in this field.

Suggested Citation

  • Matthias Meyer & Rüdiger W. Waldkirch & Michael A. Zaggl, 2012. "Relative Performance Measurement of Researchers: The Impact of Data Source Selection," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 64(4), pages 308-330, October.
    • Handle: RePEc:sbr:abstra:v:64:y:2012:i:4:p:308-330
    as

    Download full text from publisher

    File URL: http://www.vhb.de/sbr/pdfarchive.html
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bruno S. Frey, 2007. "Evaluierungen, Evaluierungen … Evaluitis," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 8(3), pages 207-220, August.
    2. Sönke Albers, 2009. "Misleading Rankings of Research in Business," German Economic Review, Verein für Socialpolitik, vol. 10(3), pages 352-363, August.
    3. Magnus Henrekson & Daniel Waldenström, 2011. "How Should Research Performance Be Measured? A Study Of Swedish Economists," Manchester School, University of Manchester, vol. 79(6), pages 1139-1156, December.
    4. Matthias Krapf, 2010. "Research evaluation and journal quality weights: Much ado about nothing?," Working Paper Series of the Department of Economics, University of Konstanz 2010-02, Department of Economics, University of Konstanz.
    5. Bruno S. Frey & Katja Rost, 2010. "Do rankings reflect research quality?," Journal of Applied Economics, Universidad del CEMA, vol. 13, pages 1-38, May.
    6. Daniel B. Klein & Eric Chiang, 2004. "The Social Science Citation Index: A Black Box—with an Ideological Bias?," Econ Journal Watch, Econ Journal Watch, vol. 1(1), pages 134-165, April.
    7. Holmstrom, Bengt & Milgrom, Paul, 1991. "Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 7(0), pages 24-52, Special I.
    8. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    9. Frey Bruno S., 2007. "Evaluierungen, Evaluierungen . . . Evaluitis," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 8(3), pages 207-220, August.
    10. Sönke Albers, 2009. "Misleading Rankings of Research in Business," German Economic Review, Verein für Socialpolitik, vol. 10(3), pages 352-363, August.
    11. Dilger, Alexander, 2009. "Rankings von Zeitschriften und Personen in der BWL," IÖB-Diskussionspapiere 5/09, University of Münster, Institute for Economic Education.
    12. Werner Reinartz, 2011. "Discussion of “Quantitative and Qualitative Rankings of Scholars”: Feeling Good or Feeling Right?," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 63(1), pages 109-114, January.
    13. Bruno S. Frey & Margit Osterloh, 2006. "Evaluations: Hidden Costs, Questionable Benefits, and Superior Alternatives," IEW - Working Papers 302, Institute for Empirical Research in Economics - University of Zurich.
    14. Bernard Raffournier & Alain Schatt, 2010. "Is European Accounting Research Fairly Reflected in Academic Journals? An Investigation of Possible Non-mainstream and Language Barrier Biases," European Accounting Review, Taylor & Francis Journals, vol. 19(1), pages 161-190.
    15. Rosenstreich, Daniela & Wooliscroft, Ben, 2009. "Measuring the impact of accounting journals using Google Scholar and the g-index," The British Accounting Review, Elsevier, vol. 41(4), pages 227-239.
    16. Diana Hicks, 1999. "The difficulty of achieving full coverage of international social science literature and the bibliometric consequences," Scientometrics, Springer;Akadémiai Kiadó, vol. 44(2), pages 193-215, February.
    17. Uschi Backes-Gellner, 2011. "Discussion of “Quantitative and Qualitative Rankings of Scholars”: Rankings upon Rankings – and no End in Sight –," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 63(1), pages 99-108, January.
    18. Thed N. Van Leeuwen & Henk F. Moed & Robert J. W. Tijssen & Martijn S. Visser & Anthony F. J. Van Raan, 2001. "Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 335-346, April.
    19. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    20. Edward P. Swanson, 2004. "Publishing in the Majors: A Comparison of Accounting, Finance, Management, and Marketing," Contemporary Accounting Research, John Wiley & Sons, vol. 21(1), pages 223-255, March.
    21. Katja Rost & Bruno S. Frey, 2011. "Quantitative and Qualitative Rankings of Scholars," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 63(1), pages 63-91, January.
    22. Soenke Albers, 2011. "Discussion of “Quantitative and Qualitative Rankings of Scholars”: Esteem Indicators: Membership in Editorial Boards or Honorary Doctorates," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 63(1), pages 92-98, January.
    23. Jim Giles, 2005. "Start your engines," Nature, Nature, vol. 438(7068), pages 554-555, December.
    24. Salvador Carmona & Isabel Gutierrez & Macario Camara, 1999. "A profile of European accounting research: evidence from leading research journals," European Accounting Review, Taylor & Francis Journals, vol. 8(3), pages 463-480.
    25. Kayvan Kousha & Mike Thelwall, 2008. "Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 273-294, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gerhard Reichmann & Christian Schlögl, 2022. "On the possibilities of presenting the research performance of an institute over a long period of time: the case of the Institute of Information Science at the University of Graz in Austria," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3193-3223, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Müller, Harry, 2012. "Die Zitationshäufigkeit als Qualitätsindikator im Rahmen der Forschungsleistungsmessung," Discussion Papers of the Institute for Organisational Economics 1/2012, University of Münster, Institute for Organisational Economics.
    2. Margit Osterloh & Bruno S. Frey, 2010. "Academic rankings and research governance," IEW - Working Papers 482, Institute for Empirical Research in Economics - University of Zurich.
    3. Agnes Bäker & Susanne Breuninger & Julia Muschallik & Kerstin Pull & Uschi Backes-Gellner, 2016. "Time to Go? (Inter)National Mobility and Appointment Success of Young Academics," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 17(3), pages 401-421, December.
    4. Margit Osterloh & Bruno S. Frey, 2009. "Research Governance in Academia: Are there Alternatives to Academic Rankings?," CREMA Working Paper Series 2009-17, Center for Research in Economics, Management and the Arts (CREMA).
    5. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    6. Müller, Harry & Dilger, Alexander, 2013. "Der Einfluss des Forschungsschwerpunkts auf den Zitationserfolg: Eine empirische Untersuchung anhand der Gesamtpublikationen deutschsprachiger Hochschullehrer für BWL," Discussion Papers of the Institute for Organisational Economics 1/2013, University of Münster, Institute for Organisational Economics.
    7. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    8. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    9. Lyudmyla Shkulipa, 2021. "Evaluation of accounting journals by coverage of accounting topics in 2018–2019," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7251-7327, September.
    10. Martin-Martin, Alberto & Orduna-Malea, Enrique & Harzing, Anne-Wil & Delgado López-Cózar, Emilio, 2017. "Can we use Google Scholar to identify highly-cited documents?," Journal of Informetrics, Elsevier, vol. 11(1), pages 152-163.
    11. Peder Olesen Larsen & Markus Ins, 2010. "The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 575-603, September.
    12. Bräuninger Michael & Haucap Justus & Muck Johannes, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 12(4), pages 339-371, November.
    13. Dilger, Alexander & Müller, Harry, 2011. "Ein Forschungsleistungsranking auf der Grundlage von Google Scholar," Discussion Papers of the Institute for Organisational Economics 12/2011, University of Münster, Institute for Organisational Economics.
    14. Judit Bar-Ilan, 2010. "Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(3), pages 495-506, March.
    15. Bornmann, Lutz & Marx, Werner & Schier, Hermann & Rahm, Erhard & Thor, Andreas & Daniel, Hans-Dieter, 2009. "Convergent validity of bibliometric Google Scholar data in the field of chemistry—Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published els," Journal of Informetrics, Elsevier, vol. 3(1), pages 27-35.
    16. Tüselmann, Heinz & Sinkovics, Rudolf R. & Pishchulov, Grigory, 2015. "Towards a consolidation of worldwide journal rankings – A classification using random forests and aggregate rating via data envelopment analysis," Omega, Elsevier, vol. 51(C), pages 11-23.
    17. Andrea Diem & Stefan C. Wolter, 2011. "The Use of Bibliometrics to Measure Research Performance in Education Sciences," Economics of Education Working Paper Series 0066, University of Zurich, Department of Business Administration (IBW), revised May 2013.
    18. Alberto Martín-Martín & Enrique Orduna-Malea & Emilio Delgado López-Cózar, 2018. "Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2175-2188, September.
    19. Christoph Endenich & Rouven Trapp, 2016. "Cooperation for Publication? An Analysis of Co-authorship Patterns in Leading Accounting Journals," European Accounting Review, Taylor & Francis Journals, vol. 25(3), pages 613-633, September.
    20. Katja Rost & Bruno S. Frey, 2011. "Quantitative and Qualitative Rankings of Scholars," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 63(1), pages 63-91, January.

    More about this item

    Keywords

    Citations; Decision Making; Google Scholar; Handelsblatt Ranking; Performance Measurement; Publications; Ranking; Research Productivity; Scopus; Social Science Citation Index;
    All these keywords.

    JEL classification:

    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • M00 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - General - - - General
    • M30 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Marketing and Advertising - - - General
    • M40 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Accounting - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sbr:abstra:v:64:y:2012:i:4:p:308-330. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: sbr (email available below). General contact details of provider: https://edirc.repec.org/data/fbmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.