IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v92y2012i2d10.1007_s11192-012-0670-4.html
   My bibliography  Save this article

The generalized propensity score methodology for estimating unbiased journal impact factors

Author

Listed:
  • Rüdiger Mutz

    (ETH Zurich)

  • Hans-Dieter Daniel

    (ETH Zurich
    University of Zurich)

Abstract

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most commonly used and prominent citation-based indicators of the performance and significance of a scientific journal. The JIF is simple, reasonable, clearly defined, and comparable over time and, what is more, can be easily calculated from data provided by Thomson Reuters, but at the expense of serious technical and methodological flaws. The paper discusses one of the core problems: The JIF is affected by bias factors (e.g., document type) that have nothing to do with the prestige or quality of a journal. For solving this problem, we suggest using the generalized propensity score methodology based on the Rubin Causal Model. Citation data for papers of all journals in the ISI subject category “Microscopy” (Journal Citation Report) are used to illustrate the proposal.

Suggested Citation

  • Rüdiger Mutz & Hans-Dieter Daniel, 2012. "The generalized propensity score methodology for estimating unbiased journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 377-390, August.
    • Handle: RePEc:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0670-4
    DOI: 10.1007/s11192-012-0670-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-012-0670-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-012-0670-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Lu, Bo & Greevy, Robert & Xu, Xinyi & Beck, Cole, 2011. "Optimal Nonbipartite Matching and Its Statistical Applications," The American Statistician, American Statistical Association, vol. 65(1), pages 21-30.
    2. Donald B. Rubin, 2004. "Teaching Statistical Inference for Causal Effects in Experiments and Observational Studies," Journal of Educational and Behavioral Statistics, , vol. 29(3), pages 343-367, September.
    3. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    4. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    5. Loet Leydesdorff & Lutz Bornmann, 2011. "How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 217-229, February.
    6. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    7. Mutz, Rüdiger & Daniel, Hans-Dieter, 2012. "Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor," Journal of Informetrics, Elsevier, vol. 6(2), pages 169-176.
    8. Christoph Neuhaus & Werner Marx & Hans‐Dieter Daniel, 2009. "The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the J," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 176-183, January.
    9. Donald B. Rubin, 1977. "Assignment to Treatment Group on the Basis of a Covariate," Journal of Educational and Behavioral Statistics, , vol. 2(1), pages 1-26, March.
    10. Elaine Zanutto & Bo Lu & Robert Hornik, 2005. "Using Propensity Score Subclassification for Multiple Treatment Doses to Evaluate a National Antidrug Media Campaign," Journal of Educational and Behavioral Statistics, , vol. 30(1), pages 59-73, March.
    11. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
    12. H. F. Moed & Th. N. Leeuwen & J. Reedijk, 1999. "Towards appropriate indicators of journal impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 575-589, November.
    13. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2012. "Evaluating continuous training programmes by using the generalized propensity score," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(2), pages 587-617, April.
    14. Loet Leydesdorff & Jung C. Shin, 2011. "How to evaluate universities in terms of their relative citation impacts: Fractional counting of citations and the normalization of differences among disciplines," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(6), pages 1146-1155, June.
    15. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2012. "Evaluating continuous training programmes by using the generalized propensity score," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(2), pages 587-617, April.
    16. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    2. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Fernando A. B. Colugnati & Sergio Firpo & Paula F. Drummond Castro & Juan E. Sepulveda & Sergio L. M. Salles-Filho, 2014. "A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 85-107, October.
    5. Xu, Shuqi & Mariani, Manuel Sebastian & Lü, Linyuan & Medo, Matúš, 2020. "Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data," Journal of Informetrics, Elsevier, vol. 14(1).
    6. Adriana Bin & Sergio Salles-Filho & Luiza Maria Capanema & Fernando Antonio Basile Colugnati, 2015. "What difference does it make? Impact of peer-reviewed scholarships on scientific production," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1167-1188, February.
    7. Pilar Valderrama & Manuel Escabias & Evaristo Jiménez-Contreras & Alberto Rodríguez-Archilla & Mariano J. Valderrama, 2018. "Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1087-1095, May.
    8. Juan Miguel Campanario, 2014. "The effect of citations on the significance of decimal places in the computation of journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(2), pages 289-298, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    3. Noémi Kreif & Richard Grieve & Iván Díaz & David Harrison, 2015. "Evaluation of the Effect of a Continuous Treatment: A Machine Learning Approach with an Application to Treatment for Traumatic Brain Injury," Health Economics, John Wiley & Sons, Ltd., vol. 24(9), pages 1213-1228, September.
    4. Mutz, Rüdiger & Daniel, Hans-Dieter, 2012. "Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor," Journal of Informetrics, Elsevier, vol. 6(2), pages 169-176.
    5. Ruth T. Chepchirchir & Ibrahim Macharia & Alice W. Murage & Charles A. O. Midega & Zeyaur R. Khan, 2017. "Impact assessment of push-pull pest management on incomes, productivity and poverty among smallholder households in Eastern Uganda," Food Security: The Science, Sociology and Economics of Food Production and Access to Food, Springer;The International Society for Plant Pathology, vol. 9(6), pages 1359-1372, December.
    6. Pilar Valderrama & Manuel Escabias & Evaristo Jiménez-Contreras & Alberto Rodríguez-Archilla & Mariano J. Valderrama, 2018. "Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1087-1095, May.
    7. Swen Kuh & Grace S. Chiu & Anton H. Westveld, 2020. "Latent Causal Socioeconomic Health Index," Papers 2009.12217, arXiv.org, revised Oct 2023.
    8. Kreif, N. & Grieve, R. & Díaz, I. & Harrison, D., 2014. "Health econometric evaluation of the effects of a continuous treatment: a machine learning approach," Health, Econometrics and Data Group (HEDG) Working Papers 14/19, HEDG, c/o Department of Economics, University of York.
    9. Hilal Atasoy & Rajiv D. Banker & Paul A. Pavlou, 2016. "On the Longitudinal Effects of IT Use on Firm-Level Employment," Information Systems Research, INFORMS, vol. 27(1), pages 6-26, March.
    10. Becker, Sascha O. & Egger, Peter H. & von Ehrlich, Maximilian, 2012. "Too much of a good thing? On the growth effects of the EU's regional policy," European Economic Review, Elsevier, vol. 56(4), pages 648-668.
    11. Ida D'Attoma & Silvia Pacei, 2018. "Evaluating the Effects of Product Innovation on the Performance of European Firms by Using the Generalised Propensity Score," German Economic Review, Verein für Socialpolitik, vol. 19(1), pages 94-112, February.
    12. Martin Huber & Yu‐Chin Hsu & Ying‐Ying Lee & Layal Lettry, 2020. "Direct and indirect effects of continuous treatments based on generalized propensity score weighting," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(7), pages 814-840, November.
    13. Carina Steckenleiter & Michael Lechner & Tim Pawlowski & Ute Schüttoff, 2023. "Do local expenditures on sports facilities affect sports participation?," Economic Inquiry, Western Economic Association International, vol. 61(4), pages 1103-1128, October.
    14. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    15. Michela Bia & Carlos A. Flores & Alfonso Flores-Lagunes & Alessandra Mattei, 2014. "A Stata package for the application of semiparametric estimators of dose–response functions," Stata Journal, StataCorp LLC, vol. 14(3), pages 580-604, September.
    16. Steckenleiter, Carina & Lechner, Michael & Pawlowski, Tim & Schüttoff, Ute, 2019. "Do local public expenditures on sports facilities affect sports participation in Germany?," Economics Working Paper Series 1905, University of St. Gallen, School of Economics and Political Science.
    17. Flores, Carlos A. & Mitnik, Oscar A., 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," IZA Discussion Papers 4451, Institute of Labor Economics (IZA).
    18. A. Giffin & B. J. Reich & S. Yang & A. G. Rappold, 2023. "Generalized propensity score approach to causal inference with spatial interference," Biometrics, The International Biometric Society, vol. 79(3), pages 2220-2231, September.
    19. Peter H. Egger & Maximilian von Ehrlich & Douglas R. Nelson, 2012. "Migration and Trade," The World Economy, Wiley Blackwell, vol. 35(2), pages 216-241, February.
    20. repec:lic:licosd:29311 is not listed on IDEAS
    21. Emiliano Magrini & Pierluigi Montalbano & Silvia Nenci, 2013. "Are the EU trade preferences really effective? A Generalized Propensity Score evaluation of the Southern Mediterranean Countries' case in agriculture and fishery," Working Papers 2/13, Sapienza University of Rome, DISS.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0670-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.