IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v8y2014i1p181-196.html
   My bibliography  Save this article

Estimating the accuracies of journal impact factor through bootstrap

Author

Listed:
  • Chen, Kuan-Ming
  • Jen, Tsung-Hau
  • Wu, Margaret

Abstract

The journal impact factor (JIF) reported in journal citation reports has been used to represent the influence and prestige of a journal. Whereas the consideration of the stochastic nature of a statistic is a prerequisite for statistical inference, the estimation of JIF uncertainty is necessary yet unavailable for comparing the impact among journals. Using journals in the Database of Research in Science Education (DoRISE), the current study proposes bootstrap methods to estimate the JIF variability. The paper also provides a comprehensive exposition of the sources of JIF variability. The collections of articles in the year of interest and in the preceding years both contribute to JIF variability. In addition, the variability estimate differs depending on the way a database selects its journals for inclusion. In the bootstrap process, the nested structure of articles in a journal was accounted for to ensure that each bootstrap replication reflects the actual citation characteristics of articles in the journal. In conclusion, the proposed point and interval estimates of the JIF statistic are obtained and more informative inferences on the impact of journals can be drawn.

Suggested Citation

  • Chen, Kuan-Ming & Jen, Tsung-Hau & Wu, Margaret, 2014. "Estimating the accuracies of journal impact factor through bootstrap," Journal of Informetrics, Elsevier, vol. 8(1), pages 181-196.
    • Handle: RePEc:eee:infome:v:8:y:2014:i:1:p:181-196
    DOI: 10.1016/j.joi.2013.11.007
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157713001089
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2013.11.007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Frederic Lord, 1965. "A strong true-score theory, with applications," Psychometrika, Springer;The Psychometric Society, vol. 30(3), pages 239-270, September.
    2. Gualberto Buela-Casal & Pandelis Perakakis & Michael Taylor & Purificación Checa, 2006. "Measuring internationality: Reflections and perspectives on academic journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 45-65, April.
    3. Ana M. Ramírez & Esther O. García & J. Antonio Del Río, 2000. "Renormalized Impact Factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(1), pages 3-9, January.
    4. Per O. Seglen, 1992. "The skewness of science," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 43(9), pages 628-638, October.
    5. Michel Zitt & Henry Small, 2008. "Modifying the journal impact factor by fractional citation weighting: The audience factor," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(11), pages 1856-1860, September.
    6. Baccini, A. & Barabesi, L. & Marcheselli, M. & Pratelli, L., 2012. "Statistical inference on the h-index with an application to top-scientist performance," Journal of Informetrics, Elsevier, vol. 6(4), pages 721-728.
    7. Schneider, Jesper W., 2013. "Caveats for using statistical significance tests in research assessments," Journal of Informetrics, Elsevier, vol. 7(1), pages 50-62.
    8. Narongrit Sombatsompop & Teerasak Markpin, 2005. "Making an equality of ISI impact factors for different subject fields," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(7), pages 676-683, May.
    9. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    10. P. Dorta-González & M. I. Dorta-González, 2013. "Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 645-672, May.
    11. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    12. Gualberto Buela-Casal & Pandelis Perakakis & Michael Taylor & Purificación Checa, 2006. "Measuring internationality: Reflections and perspectives on academic journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 45-65, April.
    13. Opthof, Tobias & Leydesdorff, Loet, 2010. "Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance," Journal of Informetrics, Elsevier, vol. 4(3), pages 423-430.
    14. Thed N. van Leeuwen & Henk F. Moed, 2002. "Development and application of journal impact measures in the Dutch science system," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 249-266, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Seiler, Christian & Wohlrabe, Klaus, 2014. "How robust are journal rankings based on the impact factor? Evidence from the economic sciences," Journal of Informetrics, Elsevier, vol. 8(4), pages 904-911.
    2. Lehmann, Robert & Wohlrabe, Klaus, 2017. "Who is the ‘Journal Grand Master’? A new ranking based on the Elo rating system," Journal of Informetrics, Elsevier, vol. 11(3), pages 800-809.
    3. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    3. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    4. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.
    5. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    6. Loet Leydesdorff, 2012. "Alternatives to the journal impact factor: I3 and the top-10% (or top-25%?) of the most-highly cited papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 355-365, August.
    7. P. Dorta-González & M. I. Dorta-González, 2013. "Comparing journals from different fields of science and social science through a JCR subject categories normalized impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 645-672, May.
    8. Tolga Yuret, 2018. "Author-weighted impact factor and reference return ratio: can we attain more equality among fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2097-2111, September.
    9. Ludo Waltman & Nees Jan Eck, 2013. "Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 699-716, September.
    10. Loet Leydesdorff, 2013. "An evaluation of impacts in “Nanoscience & nanotechnology”: steps towards standards for citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 35-55, January.
    11. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    12. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    13. A. Ferrer-Sapena & E. A. Sánchez-Pérez & L. M. González & F. Peset & R. Aleixandre-Benavent, 2015. "Mathematical properties of weighted impact factors based on measures of prestige of the citing journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2089-2108, December.
    14. Elizabeth S. Vieira & José A. N. F. Gomes, 2011. "The journal relative impact: an indicator for journal assessment," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(2), pages 631-651, November.
    15. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    16. Larivière, Vincent & Gingras, Yves, 2011. "Averages of ratios vs. ratios of averages: An empirical analysis of four levels of aggregation," Journal of Informetrics, Elsevier, vol. 5(3), pages 392-399.
    17. Tingcan Ma & Gui-Fang Wang & Ke Dong & Mukun Cao, 2012. "The Journal’s Integrated Impact Index: a new indicator for journal evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 649-658, February.
    18. B Ian Hutchins & Xin Yuan & James M Anderson & George M Santangelo, 2016. "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-25, September.
    19. Sánchez-Gil, Susana & Gorraiz, Juan & Melero-Fuentes, David, 2018. "Reference density trends in the major disciplines," Journal of Informetrics, Elsevier, vol. 12(1), pages 42-58.
    20. Dorta-González, Pablo & Dorta-González, María Isabel & Santos-Peñate, Dolores Rosa & Suárez-Vega, Rafael, 2014. "Journal topic citation potential and between-field comparisons: The topic normalized impact factor," Journal of Informetrics, Elsevier, vol. 8(2), pages 406-418.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:8:y:2014:i:1:p:181-196. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.