IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v12y2018i2p416-429.html
   My bibliography  Save this article

Ranking journals using social choice theory methods: A novel approach in bibliometrics

Author

Listed:
  • Subochev, Andrey
  • Aleskerov, Fuad
  • Pislyakov, Vladimir

Abstract

We use data on economic, management and political science journals to produce quantitative estimates of (in)consistency of the evaluations based on six popular bibliometric indicators (impact factor, 5-year impact factor, immediacy index, article influence score, SNIP and SJR). We advocate a new approach to the aggregation of journal rankings. Since the rank aggregation is a multicriteria decision problem, ranking methods from social choice theory may solve it. We apply either a direct ranking method based on the majority rule (the Copeland rule, the Markovian method) or a sorting procedure based on a tournament solution, such as the uncovered set and the minimal externally stable set. We demonstrate that the aggregate rankings reduce the number of contradictions and represent the set of the single-indicator-based rankings better than any of the six rankings themselves.

Suggested Citation

  • Subochev, Andrey & Aleskerov, Fuad & Pislyakov, Vladimir, 2018. "Ranking journals using social choice theory methods: A novel approach in bibliometrics," Journal of Informetrics, Elsevier, vol. 12(2), pages 416-429.
    • Handle: RePEc:eee:infome:v:12:y:2018:i:2:p:416-429
    DOI: 10.1016/j.joi.2018.03.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157718300919
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2018.03.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Aleskerov, Fuad, 2002. "Categories of arrovian voting schemes," Handbook of Social Choice and Welfare, in: K. J. Arrow & A. K. Sen & K. Suzumura (ed.), Handbook of Social Choice and Welfare, edition 1, volume 1, chapter 2, pages 95-129, Elsevier.
    2. John Weymark, 1984. "Arrow's theorem with social quasi-orderings," Public Choice, Springer, vol. 42(3), pages 235-246, January.
    3. Marc Fleurbaey & Peter Hammond, 2004. "Interpersonally comparable utility," Post-Print hal-00247066, HAL.
    4. Claude D'Aspremont & Louis Gevers, 1977. "Equity and the Informational Basis of Collective Choice," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 44(2), pages 199-209.
    5. Schwartz, Thomas, 1977. "Collective Choice, Separation of Issues and Vote Trading," American Political Science Review, Cambridge University Press, vol. 71(3), pages 999-1010, September.
    6. Vladimir Pislyakov, 2009. "Comparing two “thermometers”: Impact factors of 20 leading economic journals according to Journal Citation Reports and Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(3), pages 541-550, June.
    7. Éric Archambault & David Campbell & Yves Gingras & Vincent Larivière, 2009. "Comparing bibliometric statistics obtained from the Web of Science and Scopus," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(7), pages 1320-1326, July.
    8. Subochev, Andrey, 2008. "Dominant, weakly stable, uncovered sets: properties and extensions," MPRA Paper 53421, University Library of Munich, Germany.
    9. Young, H. P., 1988. "Condorcet's Theory of Voting," American Political Science Review, Cambridge University Press, vol. 82(4), pages 1231-1244, December.
    10. Smith, John H, 1973. "Aggregation of Preferences with Variable Electorate," Econometrica, Econometric Society, vol. 41(6), pages 1027-1041, November.
    11. Wade D. Cook & Tal Raviv & Alan J. Richardson, 2010. "Aggregating Incomplete Lists of Journal Rankings: An Application to Academic Accounting Journals," Accounting Perspectives, John Wiley & Sons, vol. 9(3), pages 217-235, September.
    12. Kenneth J. Arrow & Herve Raynaud, 1986. "Social Choice and Multicriterion Decision-Making," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262511754, December.
    13. I. Good, 1971. "A note on condorcet sets," Public Choice, Springer, vol. 10(1), pages 97-101, March.
    14. Wilson, Robert, 1972. "Social choice theory without the Pareto Principle," Journal of Economic Theory, Elsevier, vol. 5(3), pages 478-486, December.
    15. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    16. Fuad Aleskerov & Andrey Subochev, 2013. "Modeling optimal social choice: matrix-vector representation of various solution concepts based on majority rule," Journal of Global Optimization, Springer, vol. 56(2), pages 737-756, June.
    17. E. Garfield & I. H. Sher, 1963. "New factors in the evaluation of scientific literature through citation indexing," American Documentation, Wiley Blackwell, vol. 14(3), pages 195-201, July.
    18. Nash, John, 1950. "The Bargaining Problem," Econometrica, Econometric Society, vol. 18(2), pages 155-162, April.
    19. Fuad T. Aleskerov & Vladimir V. Pislyakov & Andrey N. Subochev, 2014. "Ranking Journals In Economics, Management And Political Science By Social Choice Theory Methods," HSE Working papers WP BRP 27/STI/2014, National Research University Higher School of Economics.
    20. Denis Bouyssou, 2004. "Monotonicity of ‘ranking by choosing’: A progress report," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 23(2), pages 249-273, October.
    21. Benjamin Ward, 1961. "Majority rule and allocation," Journal of Conflict Resolution, Peace Science Society (International), vol. 5(4), pages 379-389, December.
    22. Loet Leydesdorff, 2009. "How are new citation‐based journal indicators adding to the bibliometric toolbox?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(7), pages 1327-1336, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aleksei Y. Kondratev & Vladimir V. Mazalov, 2020. "Tournament solutions based on cooperative game theory," International Journal of Game Theory, Springer;Game Theory Society, vol. 49(1), pages 119-145, March.
    2. Tatiana Lisitskaya & Pavel Taranov & Ekaterina Ugnich & Vladimir Pislyakov, 2024. "Pillar Universities in Russia: Bibliometrics of ‘the second best’," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(1), pages 365-383, February.
    3. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    4. Saarela, Mirka & Kärkkäinen, Tommi, 2020. "Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator," Journal of Informetrics, Elsevier, vol. 14(2).
    5. Fernandez Martinez, Roberto & Lostado Lorza, Ruben & Santos Delgado, Ana Alexandra & Piedra, Nelson, 2021. "Use of classification trees and rule-based models to optimize the funding assignment to research projects: A case study of UTPL," Journal of Informetrics, Elsevier, vol. 15(1).
    6. Pislyakov, Vladimir, 2022. "On some properties of medians, percentiles, baselines, and thresholds in empirical bibliometric analysis," Journal of Informetrics, Elsevier, vol. 16(4).
    7. Mark Levene & Trevor Fenner & Judit Bar-Ilan, 2019. "Characterisation of the $$\chi$$χ-index and the rec-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 885-896, August.
    8. Shir Aviv-Reuven & Ariel Rosenfeld, 2023. "A logical set theory approach to journal subject classification analysis: intra-system irregularities and inter-system discrepancies in Web of Science and Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 157-175, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Subochev, Andrey & Zakhlebin, Igor, 2014. "Alternative versions of the global competitive industrial performance ranking constructed by methods from social choice theory," MPRA Paper 67462, University Library of Munich, Germany.
    2. Fuad Aleskerov & Andrey Subochev, 2016. "Matrix-vector representation of various solution concepts," Papers 1607.02378, arXiv.org.
    3. Subochev, A., 2016. "How Different Are the Existing Ratings of Russian Economic Journals and How to Unify Them?," Journal of the New Economic Association, New Economic Association, vol. 30(2), pages 181-192.
    4. Subochev, Andrey, 2008. "Dominant, weakly stable, uncovered sets: properties and extensions," MPRA Paper 53421, University Library of Munich, Germany.
    5. Juan Candeal, 2013. "Invariance axioms for preferences: applications to social choice theory," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 41(3), pages 453-471, September.
    6. Brandt, Felix, 2011. "Minimal stable sets in tournaments," Journal of Economic Theory, Elsevier, vol. 146(4), pages 1481-1499, July.
    7. Fuad Aleskerov & Andrey Subochev, 2013. "Modeling optimal social choice: matrix-vector representation of various solution concepts based on majority rule," Journal of Global Optimization, Springer, vol. 56(2), pages 737-756, June.
    8. Akbari, Sina & Escobedo, Adolfo R., 2023. "Beyond kemeny rank aggregation: A parameterizable-penalty framework for robust ranking aggregation with ties," Omega, Elsevier, vol. 119(C).
    9. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    10. d'Aspremont, Claude & Gevers, Louis, 2002. "Social welfare functionals and interpersonal comparability," Handbook of Social Choice and Welfare, in: K. J. Arrow & A. K. Sen & K. Suzumura (ed.), Handbook of Social Choice and Welfare, edition 1, volume 1, chapter 10, pages 459-541, Elsevier.
    11. Christian Saile & Warut Suksompong, 2020. "Robust bounds on choosing from large tournaments," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 54(1), pages 87-110, January.
    12. Kotaro Suzumura, 2002. "Introduction to social choice and welfare," Temi di discussione (Economic working papers) 442, Bank of Italy, Economic Research and International Relations Area.
    13. William M. Cockriel & James B. McDonald, 2018. "The influence of dispersion on journal impact measures," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 609-622, July.
    14. Raúl Pérez-Fernández & Bernard De Baets, 2018. "The supercovering relation, the pairwise winner, and more missing links between Borda and Condorcet," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 50(2), pages 329-352, February.
    15. Ou-Yang, Kui, 2018. "Equity, hierarchy, and ordinal social choice," Mathematical Social Sciences, Elsevier, vol. 91(C), pages 75-84.
    16. Raúl Pérez-Fernández & Bernard De Baets, 2019. "The superdominance relation, the positional winner, and more missing links between Borda and Condorcet," Journal of Theoretical Politics, , vol. 31(1), pages 46-65, January.
    17. Rabishankar Giri & Sabuj Kumar Chaudhuri, 2021. "Ranking journals through the lens of active visibility," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2189-2208, March.
    18. Noelia Rico & Camino R. Vela & Raúl Pérez-Fernández & Irene Díaz, 2021. "Reducing the Computational Time for the Kemeny Method by Exploiting Condorcet Properties," Mathematics, MDPI, vol. 9(12), pages 1-12, June.
    19. Marc Fleurbaey, 2000. "Choix social : une difficulté et de multiples possibilités," Revue Économique, Programme National Persée, vol. 51(5), pages 1215-1232.
    20. Wesley H. Holliday & Eric Pacuit, 2020. "Arrow’s decisive coalitions," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 54(2), pages 463-505, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:12:y:2018:i:2:p:416-429. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.