IDEAS home Printed from https://ideas.repec.org/a/bla/jinfst/v73y2022i8p1106-1126.html
   My bibliography  Save this article

Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluation

Author

Listed:
  • Marco Seeber
  • Jef Vlegels
  • Mattia Cattaneo

Abstract

Despite interdisciplinary research playing a pivotal role in modern science, interdisciplinary research proposals appear to have a lower chance of being funded. Scholars suggested that interdisciplinary research may be disadvantaged in the evaluation and should be earmarked specific resources and evaluated by specific panels. However, empirical evidence is limited regarding the conditions under which interdisciplinary proposals are disadvantaged. We explore this issue in the context of the European Cooperation in Science and Technology (COST) research funding framework, which, contrary to the common practice, does not organize the evaluation process around disciplinary panels, but with a panel‐free system. The sample includes data from five calls, from March 2015 to September 2017, for a total of 1,928 proposals, and 5,330 evaluations conducted by 3,050 reviewers. We find that the effect of a proposal’s degree of interdisciplinarity is negligible and not significant. We found no variation in this result across scientific fields and disciplinary expertise of reviewers, and no evidence of disciplinary “turf wars.” These results suggest that factors assumed to disadvantage interdisciplinary proposals, such as being inherently more challenging to be evaluated and being riskier, are less problematic when the evaluation is not organized around disciplinary panels but rather with a panel‐free system.

Suggested Citation

  • Marco Seeber & Jef Vlegels & Mattia Cattaneo, 2022. "Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(8), pages 1106-1126, August.
    • Handle: RePEc:bla:jinfst:v:73:y:2022:i:8:p:1106-1126
    DOI: 10.1002/asi.24617
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/asi.24617
    Download Restriction: no
    File URL: https://libkey.io/10.1002/asi.24617?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Geuna, Aldo (ed.), 2015. "Global Mobility of Research Scientists," Elsevier Monographs, Elsevier, edition 1, number 9780128013960.
    2. Geuna, Aldo & Shibayama, Sotaro, 2015. "Moving Out Of Academic Research: Why Scientists Stop Doing Research?," Department of Economics and Statistics Cognetti de Martiis LEI & BRICK - Laboratory of Economics of Innovation "Franco Momigliano", Bureau of Research in Innovation, Complexity and Knowledge, Collegio 201501, University of Turin.
    3. Anthony Hopwood, 2008. "Changing Pressures on the Research Process: On Trying to Research in an Age when Curiosity is not Enough," European Accounting Review, Taylor & Francis Journals, vol. 17(1), pages 87-96.
    4. David G Pina & Darko Hren & Ana Marušić, 2015. "Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    5. Upali W. Jayasinghe & Herbert W. Marsh & Nigel Bond, 2003. "A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 166(3), pages 279-300, October.
    6. Davide Donina & Marco Seeber & Stefano Paleari, 2017. "Inconsistencies in the Governance of Interdisciplinarity: the Case of the Italian Higher Education System," Science and Public Policy, Oxford University Press, vol. 44(6), pages 865-875.
    7. Richard Van Noorden, 2015. "Interdisciplinary research by the numbers," Nature, Nature, vol. 525(7569), pages 306-307, September.
    8. Caroline Haythornthwaite, 2006. "Learning and knowledge networks in interdisciplinary collaborations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(8), pages 1079-1092, June.
    9. Kevin J. Boudreau & Eva C. Guinan & Karim R. Lakhani & Christoph Riedl, 2016. "Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science," Management Science, INFORMS, vol. 62(10), pages 2765-2783, October.
    10. Thijs Bol & Mathijs de Vaan & Arnout van de Rijt, 2018. "The Matthew effect in science funding," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(19), pages 4887-4890, May.
    11. Juan Miguel Campanario & Erika Acedo, 2007. "Rejecting highly cited papers: The views of scientists who encounter resistance to their discoveries from other scientists," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(5), pages 734-743, March.
    12. Loet Leydesdorff, 2018. "Diversity and interdisciplinarity: how can one distinguish and recombine disparity, variety, and balance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2113-2121, September.
    13. Grit Laudel, 2006. "Conclave in the Tower of Babel: how peers review interdisciplinary research proposals," Research Evaluation, Oxford University Press, vol. 15(1), pages 57-68, April.
    14. Andy Stirling, 2007. "A General Framework for Analysing Diversity in Science, Technology and Society," SPRU Working Paper Series 156, SPRU - Science Policy Research Unit, University of Sussex Business School.
    15. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    16. Leydesdorff, Loet & Wagner, Caroline S. & Bornmann, Lutz, 2019. "Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient," Journal of Informetrics, Elsevier, vol. 13(1), pages 255-269.
    17. Liv Langfeldt, 2006. "The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments," Research Evaluation, Oxford University Press, vol. 15(1), pages 31-41, April.
    18. Lindell Bromham & Russell Dinnage & Xia Hua, 2016. "Interdisciplinary research has consistently lower funding success," Nature, Nature, vol. 534(7609), pages 684-687, June.
    19. Irwin Feller, 2002. "New organizations, old cultures: strategy and implementation of interdisciplinary programs," Research Evaluation, Oxford University Press, vol. 11(2), pages 109-116, August.
    20. Loet Leydesdorff, 2007. "Betweenness centrality as an indicator of the interdisciplinarity of scientific journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(9), pages 1303-1319, July.
    21. Wagner, Caroline S. & Roessner, J. David & Bobb, Kamau & Klein, Julie Thompson & Boyack, Kevin W. & Keyton, Joann & Rafols, Ismael & Börner, Katy, 2011. "Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature," Journal of Informetrics, Elsevier, vol. 5(1), pages 14-26.
    22. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    23. Veronica Boix Mansilla & Irwin Feller & Howard Gardner, 2006. "Quality assessment in interdisciplinary research and education," Research Evaluation, Oxford University Press, vol. 15(1), pages 69-74, April.
    24. Danielle Li, 2017. "Expertise versus Bias in Evaluation: Evidence from the NIH," American Economic Journal: Applied Economics, American Economic Association, vol. 9(2), pages 60-92, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hren, Darko & Pina, David G. & Norman, Christopher R. & Marušić, Ana, 2022. "What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports," Journal of Informetrics, Elsevier, vol. 16(2).
    2. Andrea Bonaccorsi & Nicola Melluso & Francesco Alessandro Massucci, 2022. "Exploring the antecedents of interdisciplinarity at the European Research Council: a topic modeling approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6961-6991, December.
    3. Seeber, Marco & Alon, Ilan & Pina, David G. & Piro, Fredrik Niclas & Seeber, Michele, 2022. "Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model," Technological Forecasting and Social Change, Elsevier, vol. 184(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lina Xu & Steven Dellaportas & Zhiqiang Yang & Jin Wang, 2023. "More on the relationship between interdisciplinary accounting research and citation impact," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 63(4), pages 4779-4803, December.
    2. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    3. Andrea Bonaccorsi & Nicola Melluso & Francesco Alessandro Massucci, 2022. "Exploring the antecedents of interdisciplinarity at the European Research Council: a topic modeling approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6961-6991, December.
    4. Seolmin Yang & So Young Kim, 2023. "Knowledge-integrated research is more disruptive when supported by homogeneous funding sources: a case of US federally funded research in biomedical and life sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3257-3282, June.
    5. Sander Zwanenburg & Maryam Nakhoda & Peter Whigham, 2022. "Toward greater consistency and validity in measuring interdisciplinarity: a systematic and conceptual evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7769-7788, December.
    6. Froese, Anna & Woiwode, Hendrik & Suckow, Silvio, 2019. "Mission Impossible? Neue Wege zu Interdisziplinarität: Empfehlungen für Wissenschaft, Wissenschaftspolitik und Praxis," Discussion Papers, Research Group Science Policy Studies SP III 2019-601, WZB Berlin Social Science Center.
    7. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    8. Alfonso Ávila-Robinson & Cristian Mejia & Shintaro Sengoku, 2021. "Are bibliometric measures consistent with scientists’ perceptions? The case of interdisciplinarity in research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7477-7502, September.
    9. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    10. Jingjing Ren & Fang Wang & Minglu Li, 2023. "Dynamics and characteristics of interdisciplinary research in scientific breakthroughs: case studies of Nobel-winning research in the past 120 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4383-4419, August.
    11. Qing Ke, 2023. "Interdisciplinary research and technological impact: evidence from biomedicine," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(4), pages 2035-2077, April.
    12. Shahadat Uddin & Tasadduq Imam & Mohammad Mozumdar, 2021. "Research interdisciplinarity: STEM versus non-STEM," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 603-618, January.
    13. Albert Banal-Estañol & Qianshuo Liu & Inés Macho-Stadler & David Pérez-Castrillo, 2021. "Similar-to-me Effects in the Grant Application Process: Applicants, Panelists, and the Likelihood of Obtaining Funds," Working Papers 1289, Barcelona School of Economics.
    14. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    15. Banal-Estañol, Albert & Macho-Stadler, Inés & Pérez-Castrillo, David, 2019. "Evaluation in research funding agencies: Are structurally diverse teams biased against?," Research Policy, Elsevier, vol. 48(7), pages 1823-1840.
    16. Lawson, Cornelia & Salter, Ammon, 2023. "Exploring the effect of overlapping institutional applications on panel decision-making," Research Policy, Elsevier, vol. 52(9).
    17. Yi Bu & Mengyang Li & Weiye Gu & Win‐bin Huang, 2021. "Topic diversity: A discipline scheme‐free diversity measurement for journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(5), pages 523-539, May.
    18. Stephen A Gallo & Joanne H Sullivan & Scott R Glisson, 2016. "The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications," PLOS ONE, Public Library of Science, vol. 11(10), pages 1-18, October.
    19. Xiaojing Cai & Xiaozan Lyu & Ping Zhou, 2023. "The relationship between interdisciplinarity and citation impact—a novel perspective on citation accumulation," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-12, December.
    20. Xuefeng Wang & Zhinan Wang & Ying Huang & Yun Chen & Yi Zhang & Huichao Ren & Rongrong Li & Jinhui Pang, 2017. "Measuring interdisciplinarity of a research system: detecting distinction between publication categories and citation categories," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 2023-2039, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jinfst:v:73:y:2022:i:8:p:1106-1126. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.asis.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.