IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v31y2022i4p486-497..html
   My bibliography  Save this article

Making the cut: How panel reviewers use evaluation devices to select applications at the European Research Council

Author

Listed:
  • Lucas Brunet
  • Ruth Müller

Abstract

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a set of evaluation devices that offer pragmatic and standardized ways of evaluating in a time-constrained and highly competitive setting. Through the use of evaluation devices, panel reviewers enact and generate a distinct reviewing expertise that encompasses subject-specific knowledge and knowledge about how to accomplish evaluation within a situated setting. We find that ERC panel reviewers employ four evaluation devices during the first step of ERC reviews: first, reviewers base judgments on applicants’ prior achievements (delegation devices); second, they adjust their evaluations of individual applications to the quality of a given set of applications (calibration devices); third, they combine multiple elements to assess the feasibility of proposals (articulation devices); and finally, they consider the impact of the proposed research on science and society (contribution devices). We show that the current use of these devices generates what we have termed evaluative pragmatism: a mode of reviewing that is shaped by and accommodated to the need to review many high-quality proposals in a short time period with possibly limited expert knowledge. In conclusion, we discuss how the prevalence of evaluative pragmatism in the first step of ERC panel reviews shapes candidate selection, particularly regarding human and epistemic diversity in European research.

Suggested Citation

  • Lucas Brunet & Ruth Müller, 2022. "Making the cut: How panel reviewers use evaluation devices to select applications at the European Research Council," Research Evaluation, Oxford University Press, vol. 31(4), pages 486-497.
    • Handle: RePEc:oup:rseval:v:31:y:2022:i:4:p:486-497.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvac040
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Pleun van Arensbergen & Inge van der Weijden & Peter van den Besselaar, 2014. "Different views on scholarly talent: What are the talents we are looking for in science?," Research Evaluation, Oxford University Press, vol. 23(4), pages 273-284.
    2. Ruth Müller & Sarah de Rijcke, 2017. "Exploring the epistemic impacts of academic performance indicators in the life sciences," Research Evaluation, Oxford University Press, vol. 26(3), pages 157-168.
    3. Terttu Luukkonen, 2014. "The European Research Council and the European research funding landscape," Science and Public Policy, Oxford University Press, vol. 41(1), pages 29-43.
    4. Ruth Müller & Sarah de Rijcke, 2017. "Thinking with Indicators. Exploring the Epistemic Impacts of Academic Performance Indicators in the Life Sciences," Research Evaluation, Oxford University Press, vol. 26(4), pages 361-361.
    5. Laudel, Grit & Gläser, Jochen, 2014. "Beyond breakthrough research: Epistemic properties of research and their consequences for research funding," Research Policy, Elsevier, vol. 43(7), pages 1204-1216.
    6. Fabian Muniesa & Yuval Millo & Michel Callon, 2007. "An introduction to market devices," Post-Print halshs-00177928, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tóth, Tamás & Demeter, Márton & Csuhai, Sándor & Major, Zsolt Balázs, 2024. "When career-boosting is on the line: Equity and inequality in grant evaluation, productivity, and the educational backgrounds of Marie Skłodowska-Curie Actions individual fellows in social sciences an," Journal of Informetrics, Elsevier, vol. 18(2).
    2. Tony Ross-Hellauer & Thomas Klebel & Petr Knoth & Nancy Pontika, 2024. "Value dissonance in research(er) assessment: individual and perceived institutional priorities in review, promotion, and tenure," Science and Public Policy, Oxford University Press, vol. 51(3), pages 337-351.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:plo:pcbi00:1006036 is not listed on IDEAS
    2. repec:plo:pone00:0221212 is not listed on IDEAS
    3. Alexander Schniedermann, 2021. "A comparison of systematic reviews and guideline-based systematic reviews in medical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9829-9846, December.
    4. Lai Ma, 2023. "Information, platformized," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(2), pages 273-282, February.
    5. Michaela Strinzel & Josh Brown & Wolfgang Kaltenbrunner & Sarah Rijcke & Michael Hill, 2021. "Ten ways to improve academic CVs for fairer research assessment," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 8(1), pages 1-4, December.
    6. Eugenio Petrovich, 2022. "Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2195-2233, May.
    7. Frank J. Rijnsoever & Laurens K. Hessels, 2021. "How academic researchers select collaborative research projects: a choice experiment," The Journal of Technology Transfer, Springer, vol. 46(6), pages 1917-1948, December.
    8. Gabriel-Alexandru Vîiu & Mihai Păunescu, 2021. "The citation impact of articles from which authors gained monetary rewards based on journal metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4941-4974, June.
    9. Fabio Zagonari, 2019. "Scientific Production and Productivity for Characterizing an Author’s Publication History: Simple and Nested Gini’s and Hirsch’s Indexes Combined," Publications, MDPI, vol. 7(2), pages 1-30, May.
    10. Gabriel-Alexandru Vȋiu & Mihai Păunescu, 2021. "The lack of meaningful boundary differences between journal impact factor quartiles undermines their independent use in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1495-1525, February.
    11. Claartje J Vinkenburg & Sara Connolly & Stefan Fuchs & Channah Herschberg & Brigitte Schels, 2020. "Mapping career patterns in research: A sequence analysis of career histories of ERC applicants," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-19, July.
    12. Sandra Rousseau & Ronald Rousseau, 2021. "Bibliometric Techniques And Their Use In Business And Economics Research," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1428-1451, December.
    13. Piotr Śpiewanowski & Oleksandr Talavera, 2021. "Journal rankings and publication strategy," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3227-3242, April.
    14. Bear, Laura, 2020. "Speculations on infrastructure: from colonial public works to a postcolonial global asset class on the Indian Railways 1840-2017," LSE Research Online Documents on Economics 103445, London School of Economics and Political Science, LSE Library.
    15. Thomas Wainwright, 2011. "Elite Knowledges: Framing Risk and the Geographies of Credit," Environment and Planning A, , vol. 43(3), pages 650-665, March.
    16. Rezende, Daniel Carvalho de, . "Politics in Food Markets: alternative modes of qualification and engaging," Brazilian Journal of Rural Economy and Sociology (Revista de Economia e Sociologia Rural-RESR), Sociedade Brasileira de Economia e Sociologia Rural, vol. 52(2), pages 1-14.
    17. Nick Bernards, 2019. "Tracing mutations of neoliberal development governance: ‘Fintech’, failure and the politics of marketization," Environment and Planning A, , vol. 51(7), pages 1442-1459, October.
    18. Gerhard Rainer, 2021. "Geographies of qualification in the global fine wine market," Environment and Planning A, , vol. 53(1), pages 95-112, February.
    19. Alaric Bourgoin & Nicolas Bencherki, 2013. "The performance of authority in organizations: an example from management consulting," CSI Working Papers Series 029, Centre de Sociologie de l'Innovation (CSI), Mines ParisTech.
    20. Faulconbridge, James R. & Muzio, Daniel, 2021. "Valuation devices and the dynamic legitimacy-performativity nexus: The case of PEP in the English legal profession," Accounting, Organizations and Society, Elsevier, vol. 91(C).
    21. Conor O’Kane & Jing A. Zhang & Jarrod Haar & James A. Cunningham, 2023. "How scientists interpret and address funding criteria: value creation and undesirable side effects," Small Business Economics, Springer, vol. 61(2), pages 799-826, August.
    22. Julian Hamann & Frerk Blome & Anna Kosmützky, 2022. "Devices of evaluation: Institutionalization and impact—Introduction to the special issue," Research Evaluation, Oxford University Press, vol. 31(4), pages 423-428.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:31:y:2022:i:4:p:486-497.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.