IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v130y2025i4d10.1007_s11192-025-05255-9.html
   My bibliography  Save this article

Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy

Author

Listed:
  • Alberto Baccini

    (Università degli Studi di Siena)

  • Cristina Re

    (Università degli Studi di Siena)

Abstract

Research evaluation is usually governed by panels of peers. Procedural fairness refers to the principles that ensures decisions are made through a fair and transparent process. It requires that the composition of panels is fair. A fair panel is usually defined in terms of observable characteristics of scholars such as gender or affiliations. The formal adherence to these criteria is not sufficient to guarantee a fair composition in terms of scholarly thinking, background, or policy orientation. An empirical strategy for exploring the fairness in the intellectual composition of panels is proposed, based on the observation of links between panellists. The case study regards the three panels selected to evaluate research in economics, statistics and business during the Italian research assessment exercises. The first two panels were appointed directly by the governmental agency responsible for the evaluation, while the third was randomly selected. Hence the third panel can be considered as a control for evaluating about the fairness of the others. The fair representation is explored by comparing the networks of panellists based on their co-authorship relations, the networks based on journals in which they published and the networks based on their affiliated institutions (universities, research centres and newspapers). The results show that the members of the first two panels had connections much higher than the members of the control group. Hence the composition of the first two panels should be considered as unfair, as the results of the research assessments.

Suggested Citation

  • Alberto Baccini & Cristina Re, 2025. "Is the panel fair? Evaluating panel compositions through network analysis. The case of research assessments in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(4), pages 2093-2135, April.
    • Handle: RePEc:spr:scient:v:130:y:2025:i:4:d:10.1007_s11192-025-05255-9
    DOI: 10.1007/s11192-025-05255-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-025-05255-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-025-05255-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. James J. Heckman & Sidharth Moktan, 2020. "Publishing and Promotion in Economics: The Tyranny of the Top Five," Journal of Economic Literature, American Economic Association, vol. 58(2), pages 419-470, June.
    2. Christine Wennerås & Agnes Wold, 1997. "Nepotism and sexism in peer-review," Nature, Nature, vol. 387(6631), pages 341-343, May.
    3. Esaiasson, Peter & Persson, Mikael & Gilljam, Mikael & Lindholm, Torun, 2019. "Reconsidering the Role of Procedures for Decision Acceptance," British Journal of Political Science, Cambridge University Press, vol. 49(1), pages 291-314, January.
    4. Baccini, Federica & Barabesi, Lucio & Baccini, Alberto & Khelfaoui, Mahdi & Gingras, Yves, 2022. "Similarity network fusion for scholarly journals," Journal of Informetrics, Elsevier, vol. 16(1).
    5. Bruno Frey & Matthias Benz & Alois Stutzer, 2004. "Introducing Procedural Utility: Not Only What, but Also How Matters," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 160(3), pages 377-401, September.
    6. Marcella Corsi & Carlo D'Ippoliti & Federico Lucidi, 2010. "Pluralism at Risk? Heterodox Economic Approaches and the Evaluation of Economic Research in Italy," American Journal of Economics and Sociology, Wiley Blackwell, vol. 69(5), pages 1495-1529, November.
    7. Frederic S. Lee, 2007. "The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 31(2), pages 309-325, March.
    8. Liv Langfeldt, 2004. "Expert panels evaluating research: decision-making and sources of bias," Research Evaluation, Oxford University Press, vol. 13(1), pages 51-62, April.
    9. Marcella Corsi & Carlo D'Ippoliti & Federico Lucidi, 2011. "On the Evaluation of Economic Research: The Case of Italy," Economia politica, Società editrice il Mulino, issue 3, pages 369-402.
    10. Alberto Baccini & Lucio Barabesi & Giuseppe De Nicolao, 2020. "On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-28, November.
    11. Frederic S. Lee & Xuan Pham & Gyun Gu, 2013. "The UK Research Assessment Exercise and the narrowing of UK economics," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 37(4), pages 693-717.
    12. Corsi, Marcella & D’Ippoliti, Carlo & Zacchia, Giulia, 2019. "Diversity of backgrounds and ideas: The case of research evaluation in economics," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    13. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    14. A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.
    15. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Corsi, Marcella & D’Ippoliti, Carlo & Zacchia, Giulia, 2019. "Diversity of backgrounds and ideas: The case of research evaluation in economics," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    2. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    3. Johansson, Dan & Karlsson, Johan & Malm, Arvid, 2020. "Family business—A missing link in economics?," Journal of Family Business Strategy, Elsevier, vol. 11(1).
    4. Matthias Aistleitner & Jakob Kapeller & Stefan Steinerberger, 2018. "Citation Patterns in Economics and Beyond," Working Papers Series 85, Institute for New Economic Thinking.
    5. Lorenzo Ductor & Bauke Visser, 2023. "Concentration of power at the editorial boards of economics journals," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 189-238, April.
    6. Marcella Corsi & Carlo D'Ippoliti & Federico Lucidi, 2011. "On the Evaluation of Economic Research: The Case of Italy," Economia politica, Società editrice il Mulino, issue 3, pages 369-402.
    7. William A. Jackson, 2018. "Strategic Pluralism and Monism in Heterodox Economics," Review of Radical Political Economics, Union for Radical Political Economics, vol. 50(2), pages 237-251, June.
    8. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    9. Ferenc Moksony & Rita Hegedűs & Melinda Császár, 2014. "Rankings, research styles, and publication cultures: a study of American sociology departments," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(3), pages 1715-1729, December.
    10. Önder, Ali Sina & Schweitzer, Sascha & Yilmazkuday, Hakan, 2021. "Specialization, field distance, and quality in economists’ collaborations," Journal of Informetrics, Elsevier, vol. 15(4).
    11. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    12. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    13. Alberto Baccini & Lucio Barabesi & Giuseppe De Nicolao, 2020. "On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-28, November.
    14. Bernard Chavance & Agnès Labrousse, 2018. "Institutions and ‘Science’: The Contest about Pluralism in Economics in France," Review of Political Economy, Taylor & Francis Journals, vol. 30(2), pages 190-209, April.
    15. Fabian Scheidegger & Andre Briviba & Bruno S. Frey, 2023. "Behind the curtains of academic publishing: strategic responses of economists and business scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4765-4790, August.
    16. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    17. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    18. Heise, Arne, 2019. "Ideology and pluralism: A German view," ZÖSS-Discussion Papers 75, University of Hamburg, Centre for Economic and Sociological Studies (CESS/ZÖSS).
    19. Slavica Manic, PhD, 2014. "Has Economics Lost Its Own Identity?," Asian Economic and Financial Review, Asian Economic and Social Society, vol. 4(9), pages 1190-1200, September.
    20. Li, Meiling & Wang, Yang & Du, Haifeng & Bai, Aruhan, 2024. "Motivating innovation: The impact of prestigious talent funding on junior scientists," Research Policy, Elsevier, vol. 53(9).

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:130:y:2025:i:4:d:10.1007_s11192-025-05255-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.