IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v9y2015i4p704-721.html
   My bibliography  Save this article

Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters

Author

Listed:
  • Rahman, A.I.M. Jakaria
  • Guns, Raf
  • Rousseau, Ronald
  • Engels, Tim C.E.

Abstract

Discipline-specific research evaluation exercises are typically carried out by panels of peers, known as expert panels. To the best of our knowledge, no methods are available to measure overlap in expertise between an expert panel and the units under evaluation. This paper explores bibliometric approaches to determine this overlap, using two research evaluations of the departments of Chemistry (2009) and Physics (2010) of the University of Antwerp as a test case. We explore the usefulness of overlay mapping on a global map of science (with Web of Science subject categories) to gauge overlap of expertise and introduce a set of methods to determine an entity's barycenter according to its publication output. Barycenters can be calculated starting from a similarity matrix of subject categories (N dimensions) or from a visualization thereof (2 dimensions). We compare the results of the N-dimensional method with those of two 2-dimensional ones (Kamada–Kawai maps and VOS maps) and find that they yield very similar results. The distance between barycenters is used as an indicator of expertise overlap. The results reveal that there is some discrepancy between the panel's and the groups’ publications in both the Chemistry and the Physics departments. The panels were not as diverse as the groups that were assessed. The match between the Chemistry panel and the Department was better than that between the Physics panel and the Department.

Suggested Citation

  • Rahman, A.I.M. Jakaria & Guns, Raf & Rousseau, Ronald & Engels, Tim C.E., 2015. "Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters," Journal of Informetrics, Elsevier, vol. 9(4), pages 704-721.
    • Handle: RePEc:eee:infome:v:9:y:2015:i:4:p:704-721
    DOI: 10.1016/j.joi.2015.07.009
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157715000553
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2015.07.009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Verleysen, Frederik T. & Engels, Tim C.E., 2014. "Barycenter representation of book publishing internationalization in the Social Sciences and Humanities," Journal of Informetrics, Elsevier, vol. 8(1), pages 234-240.
    2. Qiuju Zhou & Ronald Rousseau & Liying Yang & Ting Yue & Guoliang Yang, 2012. "A general framework for describing diversity within systems and similarity between systems with applications in informetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 787-812, December.
    3. N Rons & A De Bruyn & J Cornelis, 2008. "Research evaluation per discipline: a peer-review method and its outcomes," Research Evaluation, Oxford University Press, vol. 17(1), pages 45-57, March.
    4. Loet Leydesdorff & Ismael Rafols & Chaomei Chen, 2013. "Interactive overlays of journals and the measurement of interdisciplinarity on the basis of aggregated journal–journal citations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(12), pages 2573-2586, December.
    5. Ismael Rafols & Alan L. Porter & Loet Leydesdorff, 2010. "Science overlay maps: A new tool for research policy and library management," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(9), pages 1871-1887, September.
    6. Thed N. van Leeuwen & Clara Calero Medina, 2012. "Redefining the field of economics: Improving field normalization for the application of bibliometric techniques in the field of economics," Research Evaluation, Oxford University Press, vol. 21(1), pages 61-70, February.
    7. Nees Jan van Eck & Ludo Waltman & Rommert Dekker & Jan van den Berg, 2010. "A comparison of two techniques for bibliometric mapping: Multidimensional scaling and VOS," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(12), pages 2405-2416, December.
    8. Loet Leydesdorff & Stephen Carley & Ismael Rafols, 2013. "Global maps of science based on the new Web-of-Science categories," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 589-593, February.
    9. Liv Langfeldt, 2004. "Expert panels evaluating research: decision-making and sources of bias," Research Evaluation, Oxford University Press, vol. 13(1), pages 51-62, April.
    10. Loet Leydesdorff & Ismael Rafols, 2009. "A global map of science based on the ISI subject categories," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(2), pages 348-362, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rousseau, Ronald & Guns, Raf & Rahman, A.I.M. Jakaria & Engels, Tim C.E., 2017. "Measuring cognitive distance between publication portfolios," Journal of Informetrics, Elsevier, vol. 11(2), pages 583-594.
    2. Hongyu Zhou & Raf Guns & Tim C. E. Engels, 2022. "Are social sciences becoming more interdisciplinary? Evidence from publications 1960–2014," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(9), pages 1201-1221, September.
    3. A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.
    4. Rons, Nadine, 2018. "Bibliometric approximation of a scientific specialty by combining key sources, title words, authors and references," Journal of Informetrics, Elsevier, vol. 12(1), pages 113-132.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.
    2. Andrea Bonaccorsi & Nicola Melluso & Francesco Alessandro Massucci, 2022. "Exploring the antecedents of interdisciplinarity at the European Research Council: a topic modeling approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6961-6991, December.
    3. Rousseau, Ronald & Guns, Raf & Rahman, A.I.M. Jakaria & Engels, Tim C.E., 2017. "Measuring cognitive distance between publication portfolios," Journal of Informetrics, Elsevier, vol. 11(2), pages 583-594.
    4. Shiji Chen & Clément Arsenault & Yves Gingras & Vincent Larivière, 2015. "Exploring the interdisciplinary evolution of a discipline: the case of Biochemistry and Molecular Biology," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1307-1323, February.
    5. Ying Huang & Wolfgang Glänzel & Lin Zhang, 2021. "Tracing the development of mapping knowledge domains," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6201-6224, July.
    6. Jielan Ding & Per Ahlgren & Liying Yang & Ting Yue, 2018. "Disciplinary structures in Nature, Science and PNAS: journal and country levels," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1817-1852, September.
    7. Nieminen, Paavo & Pölönen, Ilkka & Sipola, Tuomo, 2013. "Research literature clustering using diffusion maps," Journal of Informetrics, Elsevier, vol. 7(4), pages 874-886.
    8. Yuxian Liu & Ewelina Biskup & Yueqian Wang & Fengfeng Cai & Xiaoyan Zhang, 2020. "A new territory and its pioneer: opening up a dominant research stream for a translational research area," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1213-1228, November.
    9. Sarin, Shikhar & Haon, Christophe & Belkhouja, Mustapha & Mas-Tur, Alicia & Roig-Tierno, Norat & Sego, Trina & Porter, Alan & Merigó, José M. & Carley, Stephen, 2020. "Uncovering the knowledge flows and intellectual structures of research in Technological Forecasting and Social Change: A journey through history," Technological Forecasting and Social Change, Elsevier, vol. 160(C).
    10. Xiaozan Lyu & Ping Zhou & Loet Leydesdorff, 2020. "Eco-system mapping of techno-science linkages at the level of scholarly journals and fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2037-2055, September.
    11. T. Gorjiara & C. Baldock, 2014. "Nanoscience and nanotechnology research publications: a comparison between Australia and the rest of the world," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(1), pages 121-148, July.
    12. Sun, Xiaoling & Ding, Kun & Lin, Yuan, 2016. "Mapping the evolution of scientific fields based on cross-field authors," Journal of Informetrics, Elsevier, vol. 10(3), pages 750-761.
    13. Carlos Olmeda-Gómez & Carlos Romá-Mateo & Maria-Antonia Ovalle-Perandones, 2019. "Overview of trends in global epigenetic research (2009–2017)," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1545-1574, June.
    14. Ying Huang & Donghua Zhu & Qi Lv & Alan L. Porter & Douglas K. R. Robinson & Xuefeng Wang, 2017. "Early insights on the Emerging Sources Citation Index (ESCI): an overlay map-based bibliometric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 2041-2057, June.
    15. Seokbeom Kwon & Alan Porter & Jan Youtie, 2016. "Navigating the innovation trajectories of technology by combining specialization score analyses for publications and patents: graphene and nano-enabled drug delivery," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 1057-1071, March.
    16. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    17. Leydesdorff, Loet & Rafols, Ismael, 2012. "Interactive overlays: A new method for generating global journal maps from Web-of-Science data," Journal of Informetrics, Elsevier, vol. 6(2), pages 318-332.
    18. Cecere, Grazia & Martinelli, Arianna, 2017. "Drivers of knowledge accumulation in electronic waste management: An analysis of publication data," Research Policy, Elsevier, vol. 46(5), pages 925-938.
    19. Jungjoon Kim & Sangpil Lee & We Shim & Jongseok Kang, 2016. "A Mapping of Marine Biodiversity Research Trends and Collaboration in the East Asia Region from 1996–2015," Sustainability, MDPI, vol. 8(10), pages 1-17, October.
    20. Stephen Carley & Alan L. Porter, 2012. "A forward diversity index," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 407-427, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:9:y:2015:i:4:p:704-721. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.