IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v179y2016i1p1-63.html
   My bibliography  Save this article

Statistical modelling of citation exchange between statistics journals

Author

Listed:
  • Cristiano Varin
  • Manuela Cattelan
  • David Firth

Abstract

type="main" xml:id="rssa12124-abs-0001"> Rankings of scholarly journals based on citation data are often met with scepticism by the scientific community. Part of the scepticism is due to disparity between the common perception of journals’ prestige and their ranking based on citation counts. A more serious concern is the inappropriate use of journal rankings to evaluate the scientific influence of researchers. The paper focuses on analysis of the table of cross-citations among a selection of statistics journals. Data are collected from the Web of Science database published by Thomson Reuters. Our results suggest that modelling the exchange of citations between journals is useful to highlight the most prestigious journals, but also that journal citation data are characterized by considerable heterogeneity, which needs to be properly summarized. Inferential conclusions require care to avoid potential overinterpretation of insignificant differences between journal ratings. Comparison with published ratings of institutions from the UK's research assessment exercise shows strong correlation at aggregate level between assessed research quality and journal citation ‘export scores’ within the discipline of statistics.

Suggested Citation

  • Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    • Handle: RePEc:bla:jorssa:v:179:y:2016:i:1:p:1-63
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1111/rssa.2016.179.issue-1
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    2. Loet Leydesdorff & Filippo Radicchi & Lutz Bornmann & Claudio Castellano & Wouter Nooy, 2013. "Field-normalized impact factors (IFs): A comparison of rescaling and fractionally counted IFs," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(11), pages 2299-2309, November.
    3. Theoharakis V. & Skordia M., 2003. "How Do Statisticians Perceive Statistics Journals?," The American Statistician, American Statistical Association, vol. 57, pages 115-123, May.
    4. Loet Leydesdorff & Tobias Opthof, 2010. "Scopus's source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(11), pages 2365-2369, November.
    5. Kevin W. Boyack & Richard Klavans & Katy Börner, 2005. "Mapping the backbone of science," Scientometrics, Springer;Akadémiai Kiadó, vol. 64(3), pages 351-374, August.
    6. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    7. Klaus Ritzberger, 2008. "A Ranking of Journals in Economics and Related Fields," German Economic Review, Verein für Socialpolitik, vol. 9, pages 402-430, November.
    8. Mark E. Glickman, 1999. "Parameter Estimation in Large Dynamic Paired Comparison Experiments," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 48(3), pages 377-394.
    9. Turner, Heather & Firth, David, 2012. "Bradley-Terry Models in R: The BradleyTerry2 Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i09).
    10. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    11. Fionn Murtagh & Michael J. Kurtz, 2016. "The Classification Society’s Bibliography Over Four Decades: History and Content Analysis," Journal of Classification, Springer;The Classification Society, vol. 33(1), pages 6-29, April.
    12. Gelman A. & Pasarica C. & Dodhia R., 2002. "Lets Practice What We Preach: Turning Tables into Graphs," The American Statistician, American Statistical Association, vol. 56, pages 121-130, May.
    13. Bien, Jacob & Tibshirani, Robert, 2011. "Hierarchical Clustering With Prototypes via Minimax Linkage," Journal of the American Statistical Association, American Statistical Association, vol. 106(495), pages 1075-1084.
    14. Hoff P.D. & Raftery A.E. & Handcock M.S., 2002. "Latent Space Approaches to Social Network Analysis," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 1090-1098, December.
    15. Pablo D. Batista & Mônica G. Campiteli & Osame Kinouchi, 2006. "Is it possible to compare researchers with different scientific interests?," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(1), pages 179-189, July.
    16. Frandsen, Tove Faber, 2007. "Journal self-citations—Analysing the JIF mechanism," Journal of Informetrics, Elsevier, vol. 1(1), pages 47-58.
    17. David Colquhoun, 2003. "Challenging the tyranny of impact factors," Nature, Nature, vol. 423(6939), pages 479-479, May.
    18. Luca Pratelli & Alberto Baccini & Lucio Barabesi & Marzia Marcheselli, 2012. "Statistical Analysis of the Hirsch Index," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 39(4), pages 681-694, December.
    19. Loet Leydesdorff & Lutz Bornmann, 2011. "How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 217-229, February.
    20. David J. Spiegelhalter & Nicola G. Best & Bradley P. Carlin & Angelika Van Der Linde, 2002. "Bayesian measures of model complexity and fit," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(4), pages 583-639, October.
    21. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    22. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    23. Gaines Liner & Minesh Amin, 2004. "Methods of ranking economics journals," Atlantic Economic Journal, Springer;International Atlantic Economic Society, vol. 32(2), pages 140-149, June.
    24. Ting Yan & Jinfeng Xu, 2013. "A central limit theorem in the β-model for undirected random graphs with a diverging number of vertices," Biometrika, Biometrika Trust, vol. 100(2), pages 519-524.
    25. Fan J. & Li R., 2001. "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1348-1360, December.
    26. Francesco Bartolucci & Valentino Dardanoni & Franco Peracchi, 2015. "Ranking scientific journals via latent class models for polytomous item response data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 1025-1049, October.
    27. Robert Tibshirani & Michael Saunders & Saharon Rosset & Ji Zhu & Keith Knight, 2005. "Sparsity and smoothness via the fused lasso," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(1), pages 91-108, February.
    28. Krivitsky, Pavel N. & Handcock, Mark S., 2008. "Fitting Latent Cluster Models for Networks with latentnet," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 24(i05).
    29. Moed, Henk F., 2010. "Measuring contextual citation impact of scientific journals," Journal of Informetrics, Elsevier, vol. 4(3), pages 265-277.
    30. Nowicki K. & Snijders T. A. B., 2001. "Estimation and Prediction for Stochastic Blockstructures," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1077-1087, September.
    31. Richard Van Noorden & Brendan Maher & Regina Nuzzo, 2014. "The top 100 papers," Nature, Nature, vol. 514(7524), pages 550-553, October.
    32. Donald O. Case & Georgeann M. Higgins, 2000. "How can we investigate citation behavior? A study of reasons for citing literature in communication," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(7), pages 635-645.
    33. Chen, Kuan-Ming & Jen, Tsung-Hau & Wu, Margaret, 2014. "Estimating the accuracies of journal impact factor through bootstrap," Journal of Informetrics, Elsevier, vol. 8(1), pages 181-196.
    34. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    35. Manuela Cattelan & Cristiano Varin & David Firth, 2013. "Dynamic Bradley–Terry modelling of sports tournaments," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 62(1), pages 135-150, January.
    36. Richard Van Noorden, 2013. "Brazilian citation scheme outed," Nature, Nature, vol. 500(7464), pages 510-511, August.
    37. Erjen Van Nierop, 2009. "Why do statistics journals have low impact factors?," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 63(1), pages 52-62, February.
    38. Harvey Goldstein & David J. Spiegelhalter, 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(3), pages 385-409, May.
    39. Stigler, George J & Stigler, Stephen M & Friedland, Claire, 1995. "The Journals of Economics," Journal of Political Economy, University of Chicago Press, vol. 103(2), pages 331-359, April.
    40. Loet Leydesdorff & Jung C. Shin, 2011. "How to evaluate universities in terms of their relative citation impacts: Fractional counting of citations and the normalization of differences among disciplines," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(6), pages 1146-1155, June.
    41. Zheng Tracy Ke & Jianqing Fan & Yichao Wu, 2015. "Homogeneity Pursuit," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(509), pages 175-194, March.
    42. Michel Zitt & Henry Small, 2008. "Modifying the journal impact factor by fractional citation weighting: The audience factor," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(11), pages 1856-1860, September.
    43. Manuela Cattelan & Cristiano Varin, 2013. "Hybrid Pairwise Likelihood Analysis of Animal Behavior Experiments," Biometrics, The International Biometric Society, vol. 69(4), pages 1002-1011, December.
    44. Ritzberger Klaus, 2008. "A Ranking of Journals in Economics and Related Fields," German Economic Review, De Gruyter, vol. 9(4), pages 402-430, December.
    45. Ludo Waltman & Nees Jan Eck, 2013. "Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 699-716, September.
    46. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    47. Mark P. Carpenter & Francis Narin, 1973. "Clustering of scientific journals," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 24(6), pages 425-436, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Vladimír Holý & Jan Zouhar, 2022. "Modelling time‐varying rankings with autoregressive and score‐driven dynamics," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(5), pages 1427-1450, November.
    2. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    3. Battistin, Erich & Ovidi, Marco, 2017. "Rising Stars," IZA Discussion Papers 11198, Institute of Labor Economics (IZA).
    4. Sebastian Szugat & Ivan Bakhtin & Leo Fechtel & Marc Hüsch & Julian Riehl & Carsten Tegethoff & Christine H. Müller, 2017. "Bedingungen für hohe Publikationsraten von Ländern in hochrangigen internationalen Statistik-Fachzeitschriften [Conditions for high publication rates of countries in high-ranking international stat," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 11(1), pages 33-49, April.
    5. Ke, Qing, 2018. "Comparing scientific and technological impact of biomedical research," Journal of Informetrics, Elsevier, vol. 12(3), pages 706-717.
    6. Cappelletti-Montano, Beniamino & Columbu, Silvia & Montaldo, Stefano & Musio, Monica, 2022. "Interpreting the outcomes of research assessments: A geometrical approach," Journal of Informetrics, Elsevier, vol. 16(1).
    7. Ryan P Womack, 2015. "Research Data in Core Journals in Biology, Chemistry, Mathematics, and Physics," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-22, December.
    8. Rowland G. Seymour & David Sirl & Simon P. Preston & Ian L. Dryden & Madeleine J. A. Ellis & Bertrand Perrat & James Goulding, 2022. "The Bayesian Spatial Bradley–Terry model: Urban deprivation modelling in Tanzania," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(2), pages 288-308, March.
    9. Weichen Wu & Nynke Niezink & Brian Junker, 2022. "A diagnostic framework for the Bradley–Terry model," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(S2), pages 461-484, December.
    10. Nadeem Shafique Butt & Ahmad Azam Malik & Muhammad Qaiser Shahbaz, 2021. "Bibliometric Analysis of Statistics Journals Indexed in Web of Science Under Emerging Source Citation Index," SAGE Open, , vol. 11(1), pages 21582440209, January.
    11. Francesca Giambona & Mariano Porcu & Isabella Sulis, 2017. "Students Mobility: Assessing the Determinants of Attractiveness Across Competing Territorial Areas," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 133(3), pages 1105-1132, September.
    12. François Caron & Emily B. Fox, 2017. "Sparse graphs using exchangeable random measures," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(5), pages 1295-1366, November.
    13. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    14. Yurij L. Katchanov & Yulia V. Markova, 2017. "The “space of physics journals”: topological structure and the Journal Impact Factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 313-333, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    2. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    3. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    4. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    5. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.
    6. Bornmann, Lutz & Haunschild, Robin, 2016. "Normalization of Mendeley reader impact on the reader- and paper-side: A comparison of the mean discipline normalized reader score (MDNRS) with the mean normalized reader score (MNRS) and bare reader ," Journal of Informetrics, Elsevier, vol. 10(3), pages 776-788.
    7. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    8. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    9. M. Zitt, 2011. "Behind citing-side normalization of citations: some properties of the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 329-344, October.
    10. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    11. Loet Leydesdorff & Ping Zhou & Lutz Bornmann, 2013. "How can journal impact factors be normalized across fields of science? An assessment in terms of percentile ranks and fractional counts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 96-107, January.
    12. Tolga Yuret, 2018. "Author-weighted impact factor and reference return ratio: can we attain more equality among fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2097-2111, September.
    13. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    14. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    15. Henk F. Moed, 2016. "Comprehensive indicator comparisons intelligible to non-experts: the case of two SNIP versions," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 51-65, January.
    16. Mutz, Rüdiger & Daniel, Hans-Dieter, 2012. "Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor," Journal of Informetrics, Elsevier, vol. 6(2), pages 169-176.
    17. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    18. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    19. Dorta-González, Pablo & Dorta-González, María Isabel & Santos-Peñate, Dolores Rosa & Suárez-Vega, Rafael, 2014. "Journal topic citation potential and between-field comparisons: The topic normalized impact factor," Journal of Informetrics, Elsevier, vol. 8(2), pages 406-418.
    20. Li, Yunrong & Ruiz-Castillo, Javier, 2013. "The comparison of normalization procedures based on different classification systems," Journal of Informetrics, Elsevier, vol. 7(4), pages 945-958.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:179:y:2016:i:1:p:1-63. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.