IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v32y2023i2p348-355..html
   My bibliography  Save this article

From intent to impact—The decline of broader impacts throughout an NSF project life cycle

Author

Listed:
  • Thomas Woodson
  • Sophia Boutilier

Abstract

It is important for funding agencies to evaluate if scientists accomplish their research goals. By comparing a representative sample of National Science Foundation abstracts and project outcome reports (PORs) from 2014 to 2017, this article investigates whether scientists attain the broader impacts they propose. We find that the number of broader impacts proposed in the abstracts is significantly higher than the number of broader impacts reported in the PORs. The trend is common across directorates and type of impact, except when impacts serve advantaged groups. Only the number of broader impacts for advantaged groups increases from the abstract to the POR. Despite the difference between proposed impact and reported impact, our study does not conclude that scientists are delinquent or disingenuous when they propose their research. Rather, we question the capacity of current frameworks to capture the quality of impacts and to weigh the relative importance of impacts that serve marginalized groups versus those that sustain the status quo.

Suggested Citation

  • Thomas Woodson & Sophia Boutilier, 2023. "From intent to impact—The decline of broader impacts throughout an NSF project life cycle," Research Evaluation, Oxford University Press, vol. 32(2), pages 348-355.
    • Handle: RePEc:oup:rseval:v:32:y:2023:i:2:p:348-355.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvac046
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    2. Susan E Cozzens, 2000. "Assessing federally-supported academic research in the United States," Research Evaluation, Oxford University Press, vol. 9(1), pages 5-10, April.
    3. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    2. Rosa Kuipers-Dirven & Matthijs Janssen & Jarno Hoekman, 2023. "Assessing university policies for enhancing societal impact of academic research: A multicriteria mapping approach," Research Evaluation, Oxford University Press, vol. 32(2), pages 371-383.
    3. Denis O. Gray & Harm-Jan Steenhuis, 2003. "Quantifying the benefits of participating in an industry university research center: An examination of research cost avoidance," Scientometrics, Springer;Akadémiai Kiadó, vol. 58(2), pages 281-300, October.
    4. Isabel Vogel & Chris Barnett, 2023. "Laying the Foundations for Impact: Lessons from the GCRF Evaluation," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 35(2), pages 281-297, April.
    5. García-Romero, Antonio & Escribano, Álvaro & Tribó, Josep A., 2017. "The impact of health research on length of stay in Spanish public hospitals," Research Policy, Elsevier, vol. 46(3), pages 591-604.
    6. Ohid Yaqub & Dmitry Malkov & Josh Siepel, 2023. "How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework," Research Evaluation, Oxford University Press, vol. 32(2), pages 273-285.
    7. Adrián A & Oscar Llopis & Pablo D’Este & Jordi Molas-Gallart, 2023. "Assessing the variety of collaborative practices in translational research: An analysis of scientists’ ego-networks," Research Evaluation, Oxford University Press, vol. 32(2), pages 426-440.
    8. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    9. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    10. Sabharwal, Meghna & Hu, Qian, 2013. "Participation in university-based research centers: Is it helping or hurting researchers?," Research Policy, Elsevier, vol. 42(6), pages 1301-1311.
    11. Lai Ma & Rachael Agnew, 2022. "Deconstructing impact: A framework for impact evaluation in grant applications [Evidencing Impact from Art Research: Analysis of Impact Case Studies from the REF 2014]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 289-301.
    12. Thomas, Duncan Andrew & Ramos-Vielba, Irene, 2022. "Reframing study of research(er) funding towards configurations and trails," SocArXiv uty2v, Center for Open Science.
    13. Stefan P L de Jong & Corina Balaban, 2022. "How universities influence societal impact practices: Academics’ sense-making of organizational impact strategies [Between Relevance and Excellence? Research Impact Agenda and the Production of Pol," Science and Public Policy, Oxford University Press, vol. 49(4), pages 609-620.
    14. Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    15. Hans Jonker & Florian Vanlee & Walter Ysebaert, 2022. "Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7289-7306, December.
    16. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    17. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    18. Edmundo Inácio Júnior & Cássio Garcia Ribeiro & André Tosi Furtado & Gabriela Silva, 2013. "Quao efetivas sao políticas tecnológicas de cunho impositivas? Evidencias do programa de P&D capitaneadas por uma empresa do setor elétrico brasileiro," Revista Ciencias Estratégicas, Universidad Pontificia Bolivariana, January.
    19. Samal, Parshuram & Mondal, Biswajit & Jambhulkar, Nitiprasad Namdeorao & Verma, Ramlakhan & Das, Anup Kumar & Singh, Onkar Nath, 2023. "Evaluation of crop research institutes under data and resource constraints: An alternative approach," Evaluation and Program Planning, Elsevier, vol. 97(C).
    20. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:32:y:2023:i:2:p:348-355.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.