IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v109y2016i3d10.1007_s11192-016-2132-x.html
   My bibliography  Save this article

Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level

Author

Listed:
  • A. I. M. Jakaria Rahman

    () (University of Antwerp)

  • Raf Guns

    (University of Antwerp)

  • Loet Leydesdorff

    (University of Amsterdam)

  • Tim C. E. Engels

    (University of Antwerp
    Antwerp Maritime Academy)

Abstract

Abstract When research groups are evaluated by an expert panel, it is an open question how one can determine the match between panel and research groups. In this paper, we outline two quantitative approaches that determine the cognitive distance between evaluators and evaluees, based on the journals they have published in. We use example data from four research evaluations carried out between 2009 and 2014 at the University of Antwerp. While the barycenter approach is based on a journal map, the similarity-adapted publication vector (SAPV) approach is based on the full journal similarity matrix. Both approaches determine an entity’s profile based on the journals in which it has published. Subsequently, we determine the Euclidean distance between the barycenter or SAPV profiles of two entities as an indicator of the cognitive distance between them. Using a bootstrapping approach, we determine confidence intervals for these distances. As such, the present article constitutes a refinement of a previous proposal that operates on the level of Web of Science subject categories.

Suggested Citation

  • A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.
  • Handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2132-x
    DOI: 10.1007/s11192-016-2132-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-016-2132-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pawel Sobkowicz, 2015. "Innovation Suppression and Clique Evolution in Peer-Review-Based, Competitive Research Funding Systems: An Agent-Based Model," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-13.
    2. Loet Leydesdorff & Gaston Heimeriks & Daniele Rotolo, 2016. "Journal portfolio analysis for countries, cities, and organizations: Maps and comparisons," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(3), pages 741-748, March.
    3. Verleysen, Frederik T. & Engels, Tim C.E., 2014. "Barycenter representation of book publishing internationalization in the Social Sciences and Humanities," Journal of Informetrics, Elsevier, vol. 8(1), pages 234-240.
    4. Nees Jan van Eck & Ludo Waltman & Rommert Dekker & Jan van den Berg, 2010. "A comparison of two techniques for bibliometric mapping: Multidimensional scaling and VOS," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(12), pages 2405-2416, December.
    5. Katharine Barker, 2007. "The UK Research Assessment Exercise: the evolution of a national research evaluation system," Research Evaluation, Oxford University Press, vol. 16(1), pages 3-12, March.
    6. Ludo Waltman & Nees Jan Eck, 2012. "A new methodology for constructing a publication-level classification system of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    7. Leydesdorff, Loet & Rafols, Ismael, 2012. "Interactive overlays: A new method for generating global journal maps from Web-of-Science data," Journal of Informetrics, Elsevier, vol. 6(2), pages 318-332.
    8. Nooteboom, Bart & Van Haverbeke, Wim & Duysters, Geert & Gilsing, Victor & van den Oord, Ad, 2007. "Optimal cognitive distance and absorptive capacity," Research Policy, Elsevier, vol. 36(7), pages 1016-1034, September.
    9. Richard Berendsen & Maarten Rijke & Krisztian Balog & Toine Bogers & Antal Bosch, 2013. "On the assessment of expertise profiles," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(10), pages 2024-2044, October.
    10. Lutz Bornmann & Rüdiger Mutz & Werner Marx & Hermann Schier & Hans‐Dieter Daniel, 2011. "A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(4), pages 857-879, October.
    11. N Rons & A De Bruyn & J Cornelis, 2008. "Research evaluation per discipline: a peer-review method and its outcomes," Research Evaluation, Oxford University Press, vol. 17(1), pages 45-57, March.
    12. Ismael Rafols & Alan L. Porter & Loet Leydesdorff, 2010. "Science overlay maps: A new tool for research policy and library management," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(9), pages 1871-1887, September.
    13. Peter van den Besselaar & Loet Leydesdorff, 2009. "Past performance, peer review and project selection: a case study in the social and behavioral sciences," Research Evaluation, Oxford University Press, vol. 18(4), pages 273-288, October.
    14. Cohen, Wesley M & Levinthal, Daniel A, 1989. "Innovation and Learning: The Two Faces of R&D," Economic Journal, Royal Economic Society, vol. 99(397), pages 569-596, September.
    15. Rahman, A.I.M. Jakaria & Guns, Raf & Rousseau, Ronald & Engels, Tim C.E., 2015. "Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters," Journal of Informetrics, Elsevier, vol. 9(4), pages 704-721.
    16. Kevin W. Boyack & Richard Klavans, 2014. "Creation of a highly detailed, dynamic, global model and map of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 670-685, April.
    17. Loet Leydesdorff & Ismael Rafols & Chaomei Chen, 2013. "Interactive overlays of journals and the measurement of interdisciplinarity on the basis of aggregated journal–journal citations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(12), pages 2573-2586, December.
    18. Katja Hofmann & Krisztian Balog & Toine Bogers & Maarten de Rijke, 2010. "Contextual factors for finding similar experts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(5), pages 994-1014, May.
    19. Bart Nooteboom, 2000. "Learning by Interaction: Absorptive Capacity, Cognitive Distance and Governance," Journal of Management & Governance, Springer;Accademia Italiana di Economia Aziendale (AIDEA), vol. 4(1), pages 69-92, March.
    20. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    21. Qi Wang & Ulf Sandström, 2015. "Defining the role of cognitive distance in the peer review process with an explorative study of a grant scheme in infection biology," Research Evaluation, Oxford University Press, vol. 24(3), pages 271-281.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:eee:infome:v:11:y:2017:i:2:p:583-594 is not listed on IDEAS

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2132-x. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla) or (Rebekah McClure). General contact details of provider: http://www.springer.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.