IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i4d10.1007_s11192-021-03864-8.html
   My bibliography  Save this article

Reliability of researcher capacity estimates and count data dispersion: a comparison of Poisson, negative binomial, and Conway-Maxwell-Poisson models

Author

Listed:
  • Boris Forthmann

    (University of Münster)

  • Philipp Doebler

    (TU Dortmund University)

Abstract

Item-response models from the psychometric literature have been proposed for the estimation of researcher capacity. Canonical items that can be incorporated in such models to reflect researcher performance are count data (e.g., number of publications, number of citations). Count data can be modeled by Rasch’s Poisson counts model that assumes equidispersion (i.e., mean and variance must coincide). However, the mean can be larger as compared to the variance (i.e., underdispersion), or b) smaller as compared to the variance (i.e., overdispersion). Ignoring the presence of overdispersion (underdispersion) can cause standard errors to be liberal (conservative), when the Poisson model is used. Indeed, number of publications or number of citations are known to display overdispersion. Underdispersion, however, is far less acknowledged in the literature. In the current investigation the flexible Conway-Maxwell-Poisson count model is used to examine reliability estimates of capacity in relation to various dispersion patterns. It is shown, that reliability of capacity estimates of inventors drops from .84 (Poisson) to .68 (Conway-Maxwell-Poisson) or .69 (negative binomial). Moreover, with some items displaying overdispersion and some items displaying underdispersion, the dispersion pattern in a reanalysis of Mutz and Daniel’s (2018b) researcher data was found to be more complex as compared to previous results. To conclude, a careful examination of competing models including the Conway-Maxwell-Poisson count model should be undertaken prior to any evaluation and interpretation of capacity reliability. Moreover, this work shows that count data psychometric models are well suited for decisions with a focus on top researchers, because conditional reliability estimates (i.e., reliability depending on the level of capacity) were highest for the best researchers.

Suggested Citation

  • Boris Forthmann & Philipp Doebler, 2021. "Reliability of researcher capacity estimates and count data dispersion: a comparison of Poisson, negative binomial, and Conway-Maxwell-Poisson models," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3337-3354, April.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:4:d:10.1007_s11192-021-03864-8
    DOI: 10.1007/s11192-021-03864-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-021-03864-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-021-03864-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wolfgang Glänzel & Henk F. Moed, 2013. "Opinion paper: thoughts and facts on bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 381-394, July.
    2. John C. Huber & Roland Wagner-Döbler, 2001. "Scientific production: A statistical analysis of authors in mathematical logic," Scientometrics, Springer;Akadémiai Kiadó, vol. 50(2), pages 323-337, February.
    3. John C. Huber & Roland Wagner-Döbler, 2001. "Scientific production: A statistical analysis of authors in physics, 1800-1900," Scientometrics, Springer;Akadémiai Kiadó, vol. 50(3), pages 437-453, March.
    4. Rolf Ketzler & Klaus F. Zimmermann, 2013. "A citation-analysis of economic research institutes," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(3), pages 1095-1112, June.
    5. De Boeck, Paul & Bakker, Marjan & Zwitser, Robert & Nivard, Michel & Hofman, Abe & Tuerlinckx, Francis & Partchev, Ivailo, 2011. "The Estimation of Item Response Models with the lmer Function from the lme4 Package in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 39(i12).
    6. Didegah, Fereshteh & Thelwall, Mike, 2013. "Which factors help authors produce the highest impact research? Collaboration, journal and document properties," Journal of Informetrics, Elsevier, vol. 7(4), pages 861-873.
    7. Burrell, Quentin L., 2007. "Hirsch's h-index: A stochastic model," Journal of Informetrics, Elsevier, vol. 1(1), pages 16-25.
    8. Gerhard Fischer, 1987. "Applying the principles of specific objectivity and of generalizability to the measurement of change," Psychometrika, Springer;The Psychometric Society, vol. 52(4), pages 565-587, December.
    9. Bronwyn H. Hall & Adam B. Jaffe & Manuel Trajtenberg, 2001. "The NBER Patent Citation Data File: Lessons, Insights and Methodological Tools," NBER Working Papers 8498, National Bureau of Economic Research, Inc.
    10. Subrata Chakraborty & Tomoaki Imoto, 2016. "Extended Conway-Maxwell-Poisson distribution and its properties and applications," Journal of Statistical Distributions and Applications, Springer, vol. 3(1), pages 1-19, December.
    11. M. J. Faddy & R. J. Bosch, 2001. "Likelihood-Based Modeling and Analysis of Data Underdispersed Relative to the Poisson Distribution," Biometrics, The International Biometric Society, vol. 57(2), pages 620-624, June.
    12. Seth D. Guikema & Jeremy P. Goffelt, 2008. "A Flexible Count Data Regression Model for Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 28(1), pages 213-223, February.
    13. Henk F. Moed & Gali Halevi, 2015. "Multidimensional assessment of scholarly research impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(10), pages 1988-2002, October.
    14. Li, Guan-Cheng & Lai, Ronald & D’Amour, Alexander & Doolin, David M. & Sun, Ye & Torvik, Vetle I. & Yu, Amy Z. & Fleming, Lee, 2014. "Disambiguation and co-authorship networks of the U.S. patent inventor database (1975–2010)," Research Policy, Elsevier, vol. 43(6), pages 941-955.
    15. Yutao Sun & Belle Selene Xia, 2016. "The scholarly communication of economic knowledge: a citation analysis of Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1965-1978, December.
    16. Gad Yair & Keith Goldstein, 2020. "The Annus Mirabilis paper: years of peak productivity in scientific careers," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 887-902, August.
    17. S. Chakraborty & S. H. Ong, 2016. "A COM-Poisson-type generalization of the negative binomial distribution," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 45(14), pages 4117-4135, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Boris Forthmann, 2023. "Researcher capacity estimation based on the Q model: a generalized linear mixed model perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4753-4764, August.
    2. Kennedy Ndue & Melese Mulu Baylie & Pál Goda, 2023. "Determinants of Rural Households’ Intensity of Flood Adaptation in the Fogera Rice Plain, Ethiopia: Evidence from Generalised Poisson Regression," Sustainability, MDPI, vol. 15(14), pages 1-19, July.
    3. Krzysztof Rusek & Agnieszka Kleszcz & Albert Cabellos-Aparicio, 2023. "Bayesian inference of spatial and temporal relations in AI patents for EU countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3313-3335, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chattergoon, B. & Kerr, W.R., 2022. "Winner takes all? Tech clusters, population centers, and the spatial transformation of U.S. invention," Research Policy, Elsevier, vol. 51(2).
    2. Ajay Bhaskarbhatla & Luis Cabral & Deepak Hegde & Thomas (T.L.P.R.) Peeters, 2017. "Human Capital, Firm Capabilities, and Innovation," Tinbergen Institute Discussion Papers 17-115/VII, Tinbergen Institute, revised 03 Mar 2020.
    3. Watzinger, Martin & Schnitzer, Monika, 2019. "Standing on the Shoulders of Science," Rationality and Competition Discussion Paper Series 215, CRC TRR 190 Rationality and Competition.
    4. Christian Fons-Rosen & Vincenzo Scrutinio & Katalin Szemeredi, 2016. "Colocation and knowledge diffusion: evidence from million dollar plants," CEP Discussion Papers dp1447, Centre for Economic Performance, LSE.
    5. Yang, Chia-Hsuan & Nugent, Rebecca & Fuchs, Erica R.H., 2016. "Gains from others’ losses: Technology trajectories and the global division of firms," Research Policy, Elsevier, vol. 45(3), pages 724-745.
    6. Silvestri, Daniela & Riccaboni, Massimo & Della Malva, Antonio, 2018. "Sailing in all winds: Technological search over the business cycle," Research Policy, Elsevier, vol. 47(10), pages 1933-1944.
    7. Byun, SeongK. & Fuller, Kathleen & Lin, Zhilu, 2021. "The costs and benefits associated with inventor CEOs," Journal of Corporate Finance, Elsevier, vol. 71(C).
    8. Thelwall, Mike & Fairclough, Ruth, 2015. "The influence of time and discipline on the magnitude of correlations between citation counts and quality scores," Journal of Informetrics, Elsevier, vol. 9(3), pages 529-541.
    9. Cao, Chunfang & Li, Xiaohui & Li, Xiaoyang & Zeng, Cheng & Zhou, Xuan, 2021. "Diversity and inclusion: Evidence from corporate inventors," Journal of Empirical Finance, Elsevier, vol. 64(C), pages 295-316.
    10. Filippo Mezzanotti, 2021. "Roadblock to Innovation: The Role of Patent Litigation in Corporate R&D," Management Science, INFORMS, vol. 67(12), pages 7362-7390, December.
    11. Akcigit, Ufuk & Grigsby, John & Nicholas, Tom, 2017. "The Rise of American Ingenuity: Innovation and Inventors of the Golden Age," CEPR Discussion Papers 11755, C.E.P.R. Discussion Papers.
    12. Pierre-Alexandre Balland & Cristian Jara-Figueroa & Sergio G. Petralia & Mathieu P. A. Steijn & David L. Rigby & César A. Hidalgo, 2020. "Complex economic activities concentrate in large cities," Nature Human Behaviour, Nature, vol. 4(3), pages 248-254, March.
    13. Seong K. Byun & Jong-Min & Han Xia, 2021. "Incremental vs. Breakthrough Innovation: The Role of Technology Spillovers," Management Science, INFORMS, vol. 67(3), pages 1779-1802, March.
    14. Buhr, Helena & Funk, Russell J. & Owen-Smith, Jason, 2021. "The authenticity premium: Balancing conformity and innovation in high technology industries," Research Policy, Elsevier, vol. 50(1).
    15. Hanne Peeters & Julie Callaert & Bart Looy, 2020. "Do firms profit from involving academics when developing technology?," The Journal of Technology Transfer, Springer, vol. 45(2), pages 494-521, April.
    16. Dass, Nishant & Nanda, Vikram & Xiao, Steven Chong, 2017. "Truncation bias corrections in patent data: Implications for recent research on innovation," Journal of Corporate Finance, Elsevier, vol. 44(C), pages 353-374.
    17. Daniel Bradley & Incheol Kim & Xuan Tian, 2017. "Do Unions Affect Innovation?," Management Science, INFORMS, vol. 63(7), pages 2251-2271, July.
    18. Jiang, Lin & Clark, Brent B. & Turban, Daniel B., 2023. "Overcoming the challenge of exploration: How decompartmentalization of internal communication enhances the effect of exploration on employee inventive performance," Technovation, Elsevier, vol. 119(C).
    19. David Hirshleifer & Po-Hsuan Hsu & Dongmei Li, 2018. "Innovative Originality, Profitability, and Stock Returns," The Review of Financial Studies, Society for Financial Studies, vol. 31(7), pages 2553-2605.
    20. Guan, Jiancheng & Yan, Yan & Zhang, Jing Jing, 2017. "The impact of collaboration and knowledge networks on citations," Journal of Informetrics, Elsevier, vol. 11(2), pages 407-422.

    More about this item

    Keywords

    Researcher capacity; Item response models; Rasch Poisson count model; Conway-Maxwell-Poisson count model; Dispersion; Reliability;
    All these keywords.

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:4:d:10.1007_s11192-021-03864-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.