IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v12y2018i4p1282-1295.html
   My bibliography  Save this article

The bibliometric quotient (BQ), or how to measure a researcher’s performance capacity: A Bayesian Poisson Rasch model

Author

Listed:
  • Mutz, Rüdiger
  • Daniel, Hans-Dieter

Abstract

Regarding evaluation of individual researchers, the bibliometric indicators approach has been increasingly discussed recently, but there are some problems involved with it: construct definition, measurement errors, level of scale, dimensionality, normalization. Based on a psychometric model, the Rasch model, we developed a measuring scale for the theoretical construct ‘researcher’s performance capacity,’ defined as the competency of a researcher to write influential papers. The aim was a scale that is one-dimensional and continuous, is applicable to bibliometric count variables, and takes measurement errors into account. In this paper we present the psychometric model (Bayesian Poisson Rasch model, BPR) and its assumptions and examine the behavior of the model under various sampling conditions. For a sample of N = 254 researchers in a quantitative methodology section of an undisclosed German academic society for social sciences, using the BPR model we developed a scale that we named ‘Bibliometric Quotient’ (BQ, M = 100, SD = 15) (following the term ‘intelligence quotient’). The scale fulfills most of the test-theoretical requirements (e.g., high reliability αt = .96, no differential item functioning except for academic age and German states) and in addition allows researchers to be ranked. Women’s BQ scores were 8.3 points lower on the scale than men’s BQ scores.

Suggested Citation

  • Mutz, Rüdiger & Daniel, Hans-Dieter, 2018. "The bibliometric quotient (BQ), or how to measure a researcher’s performance capacity: A Bayesian Poisson Rasch model," Journal of Informetrics, Elsevier, vol. 12(4), pages 1282-1295.
    • Handle: RePEc:eee:infome:v:12:y:2018:i:4:p:1282-1295
    DOI: 10.1016/j.joi.2018.10.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157718300798
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2018.10.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Cameron,A. Colin & Trivedi,Pravin K., 2013. "Regression Analysis of Count Data," Cambridge Books, Cambridge University Press, number 9781107667273.
    2. Lutz Bornmann & Rüdiger Mutz & Hans‐Dieter Daniel, 2008. "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 830-837, March.
    3. Margo Jansen & Marijtje Duijn, 1992. "Extensions of Rasch's multiplicative poisson model," Psychometrika, Springer;The Psychometric Society, vol. 57(3), pages 405-414, September.
    4. Todd Dewett & Angelo S. Denisi, 2004. "Exploring scholarly reputation: It's more than just productivity," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(2), pages 249-272, June.
    5. Ferdinand A. Gul & Judy S. L. Tsui, 2004. "Introduction and overview," Palgrave Macmillan Books, in: The Governance of East Asian Corporations, chapter 1, pages 1-26, Palgrave Macmillan.
    6. Hilbe,Joseph M., 2014. "Modeling Count Data," Cambridge Books, Cambridge University Press, number 9781107028333.
    7. Pedro Alvarez & Antonio Pulgarín, 1996. "The Rasch model. Measuring the impact of scientific journals: Analytical chemistry," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 47(6), pages 458-467, June.
    8. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Ryan Chute, 2009. "A Principal Component Analysis of 39 Scientific Impact Measures," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-11, June.
    9. Wim Van den Noortgate & Paul De Boeck, 2005. "Assessing and Explaining Differential Item Functioning Using Logistic Mixed Models," Journal of Educational and Behavioral Statistics, , vol. 30(4), pages 443-464, December.
    10. Lutz Bornmann & Werner Marx, 2014. "Distributions instead of single numbers: Percentiles and beam plots for the assessment of single researchers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(1), pages 206-208, January.
    11. John P. A. Ioannidis, 2011. "Fund people not projects," Nature, Nature, vol. 477(7366), pages 529-531, September.
    12. Gerhard Fischer, 1995. "Some neglected problems in IRT," Psychometrika, Springer;The Psychometric Society, vol. 60(4), pages 459-487, December.
    13. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2013. "Individual research performance: A proposal for comparing apples to oranges," Journal of Informetrics, Elsevier, vol. 7(2), pages 528-539.
    14. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    15. Tammy Harris & Joseph M. Hilbe & James W. Hardin, 2014. "Modeling count data with generalized distributions," Stata Journal, StataCorp LP, vol. 14(3), pages 562-579, September.
    16. Pedro Alvarez & Antonio Pulgarin, 1996. "The Rasch model. Measuring information from keywords: The diabetes field," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 47(6), pages 468-476, June.
    17. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    18. Sen, Amartya, 1999. "Commodities and Capabilities," OUP Catalogue, Oxford University Press, number 9780195650389.
    19. Pedro Alvarez & Antonio Pulgarín, 1997. "The diffusion of scientific journals analyzed through citations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 48(10), pages 953-958, October.
    20. Ludo Waltman & Nees Jan van Eck, 2012. "The inconsistency of the h-index," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(2), pages 406-415, February.
    21. Paola Cerchiello & Paolo Giudici, 2014. "On a statistical h index," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(2), pages 299-312, May.
    22. David J. Spiegelhalter & Nicola G. Best & Bradley P. Carlin & Angelika Linde, 2014. "The deviance information criterion: 12 years on," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(3), pages 485-493, June.
    23. Harvey Goldstein & David J. Spiegelhalter, 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(3), pages 385-409, May.
    24. Rüdiger Mutz & Hans-Dieter Daniel, 2015. "What is behind the curtain of the Leiden Ranking?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(9), pages 1950-1953, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Boris Forthmann, 2023. "Researcher capacity estimation based on the Q model: a generalized linear mixed model perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4753-4764, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chiara Bocci & Laura Grassini & Emilia Rocco, 2021. "A multiple inflated negative binomial hurdle regression model: analysis of the Italians’ tourism behaviour during the Great Recession," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 30(4), pages 1109-1133, October.
    2. Eugenio Petrovich, 2022. "Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2195-2233, May.
    3. Smith, David M. & Faddy, Malcolm J., 2016. "Mean and Variance Modeling of Under- and Overdispersed Count Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 69(i06).
    4. de Rezende, Rafael & Egert, Katharina & Marin, Ignacio & Thompson, Guilherme, 2022. "A white-boxed ISSM approach to estimate uncertainty distributions of Walmart sales," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1460-1467.
    5. Sinclair, Michael & Ghermandi, Andrea & Signorello, Giovanni & Giuffrida, Laura & De Salvo, Maria, 2022. "Valuing Recreation in Italy's Protected Areas Using Spatial Big Data," Ecological Economics, Elsevier, vol. 200(C).
    6. Andersen, Jens Peter, 2017. "An empirical and theoretical critique of the Euclidean index," Journal of Informetrics, Elsevier, vol. 11(2), pages 455-465.
    7. Loet Leydesdorff & Paul Wouters & Lutz Bornmann, 2016. "Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2129-2150, December.
    8. Fu, Xiaolan & Fu, Xiaoqing (Maggie) & Ghauri, Pervez & Hou, Jun, 2022. "International collaboration and innovation: Evidence from a leading Chinese multinational enterprise," Journal of World Business, Elsevier, vol. 57(4).
    9. Jun-You Lin, 2021. "Collaboration exploitation and exploration: does a proactive search strategy matter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8295-8329, October.
    10. Yves Fassin, 2020. "The HF-rating as a universal complement to the h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 965-990, November.
    11. Lall, Ashish, 2018. "Delays in the New York City metroplex," Transportation Research Part A: Policy and Practice, Elsevier, vol. 114(PA), pages 139-153.
    12. Lutz Bornmann & Richard Williams, 2020. "An evaluation of percentile measures of citation impact, and a proposal for making them better," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1457-1478, August.
    13. Luiz Paulo Fávero & Joseph F. Hair & Rafael de Freitas Souza & Matheus Albergaria & Talles V. Brugni, 2021. "Zero-Inflated Generalized Linear Mixed Models: A Better Way to Understand Data Relationships," Mathematics, MDPI, vol. 9(10), pages 1-28, May.
    14. Wu, Jiang & Ou, Guiyan & Liu, Xiaohui & Dong, Ke, 2022. "How does academic education background affect top researchers’ performance? Evidence from the field of artificial intelligence," Journal of Informetrics, Elsevier, vol. 16(2).
    15. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    16. Pantea Kamrani & Isabelle Dorsch & Wolfgang G. Stock, 2021. "Do researchers know what the h-index is? And how do they estimate its importance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5489-5508, July.
    17. Zhenbin Yan & Qiang Wu & Xingchen Li, 2016. "Do Hirsch-type indices behave the same in assessing single publications? An empirical study of 29 bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1815-1833, December.
    18. Maziar Montazerian & Edgar Dutra Zanotto & Hellmut Eckert, 2019. "A new parameter for (normalized) evaluation of H-index: countries as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1065-1078, March.
    19. Bornmann, Lutz & Leydesdorff, Loet, 2013. "The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000," Journal of Informetrics, Elsevier, vol. 7(2), pages 286-291.
    20. Corey J A Bradshaw & Justin M Chalker & Stefani A Crabtree & Bart A Eijkelkamp & John A Long & Justine R Smith & Kate Trinajstic & Vera Weisbecker, 2021. "A fairer way to compare researchers at any career stage and in any discipline using open-access citation data," PLOS ONE, Public Library of Science, vol. 16(9), pages 1-15, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:12:y:2018:i:4:p:1282-1295. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.