IDEAS home Printed from https://ideas.repec.org/a/oup/scippl/v49y2022i2p313-323..html
   My bibliography  Save this article

Assessing broader impacts of funded research: the US National Science Foundation v. Lamar Smith
[What is Societal Impact of Research and How Can it Be Assessed? A Literature Survey]

Author

Listed:
  • Caitlin Drummond Otten
  • Baruch Fischhoff

Abstract

The US National Science Foundation (NSF) must demonstrate the value of the research that it funds. To that end, NSF has developed two criteria for its funding decisions: intellectual merit (IM), the potential to advance knowledge, and broader impacts (BI), the potential to benefit society. Critics have argued that these criteria are insufficient, including former Rep. Lamar Smith (R-TX), who offered his own criteria delineating specific desired impacts, such as economic competitiveness. We empirically assess this criticism by comparing public ratings of NSF-funded projects, based on their Project Outcomes Report (POR), using NSF’s and Smith’s criteria. Participants rated NSF-funded research as satisfying both NSF’s and Smith’s criteria, which were moderately correlated. Adding explicit references to societal BI improved ratings slightly. Noting NSF support did not. Our results suggest that having PORs explicitly address additional criteria could increase perceived BI without compromising IM.

Suggested Citation

  • Caitlin Drummond Otten & Baruch Fischhoff, 2022. "Assessing broader impacts of funded research: the US National Science Foundation v. Lamar Smith [What is Societal Impact of Research and How Can it Be Assessed? A Literature Survey]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 313-323.
    • Handle: RePEc:oup:scippl:v:49:y:2022:i:2:p:313-323.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/scipol/scab082
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. J Britt Holbrook & Robert Frodeman, 2011. "Peer review and the ex ante assessment of societal impacts," Research Evaluation, Oxford University Press, vol. 20(3), pages 239-246, September.
    2. Daniel Sarewitz, 2011. "The dubious benefits of broader impact," Nature, Nature, vol. 475(7355), pages 141-141, July.
    3. Salter, Ammon J. & Martin, Ben R., 2001. "The economic benefits of publicly funded basic research: a critical review," Research Policy, Elsevier, vol. 30(3), pages 509-532, March.
    4. Erik Ernø-Kjølhede & Finn Hansson, 2011. "Measuring research performance during a changing relationship between science and society," Research Evaluation, Oxford University Press, vol. 20(2), pages 131-143, June.
    5. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    6. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    7. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    8. Omar Kassab, 2019. "Does public outreach impede research performance? Exploring the ‘researcher’s dilemma’ in a sustainability research center," Science and Public Policy, Oxford University Press, vol. 46(5), pages 710-720.
    9. Bozeman, Barry & Youtie, Jan, 2017. "Socio-economic impacts and public value of government-funded research: Lessons from four US National Science Foundation initiatives," Research Policy, Elsevier, vol. 46(8), pages 1387-1398.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    2. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    3. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    4. Matt, M. & Gaunand, A. & Joly, P-B. & Colinet, L., 2017. "Opening the black box of impact – Ideal-type impact pathways in a public agricultural research organization," Research Policy, Elsevier, vol. 46(1), pages 207-218.
    5. Irwin Feller, 2022. "Assessing the societal impact of publicly funded research," The Journal of Technology Transfer, Springer, vol. 47(3), pages 632-650, June.
    6. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    7. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    8. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    9. Pierre Benoit Joly & Laurence Colinet & Ariane Gaunand & Stephane Lemarié & Mireille Matt, 2016. "Agricultural research impact assessment: issues, methods and challenges," Working Papers hal-01431457, HAL.
    10. Bozeman, Barry & Youtie, Jan, 2017. "Socio-economic impacts and public value of government-funded research: Lessons from four US National Science Foundation initiatives," Research Policy, Elsevier, vol. 46(8), pages 1387-1398.
    11. de Almeida, Liliane & Augusto de Jesus Pacheco, Diego & Caten, Carla Schwengber ten & Jung, Carlos Fernando, 2021. "A methodology for identifying results and impacts in technological innovation projects," Technology in Society, Elsevier, vol. 66(C).
    12. Nathalie Taverdet-Popiolek, 2022. "Economic Footprint of a Large French Research and Technology Organisation in Europe: Deciphering a Simplified Model and Appraising the Results," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 13(1), pages 44-69, March.
    13. Pablo D’Este & Irene Ramos-Vielba & Richard Woolley & Nabil Amara, 2018. "How do researchers generate scientific and societal impacts? Toward an analytical and operational framework," Science and Public Policy, Oxford University Press, vol. 45(6), pages 752-763.
    14. Juha-Pekka Lauronen, 2022. "Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization," SAGE Open, , vol. 12(2), pages 21582440221, April.
    15. Castelnovo, Paolo & Florio, Massimo & Forte, Stefano & Rossi, Lucio & Sirtori, Emanuela, 2018. "The economic impact of technological procurement for large-scale research infrastructures: Evidence from the Large Hadron Collider at CERN," Research Policy, Elsevier, vol. 47(9), pages 1853-1867.
    16. Lutz Bornmann, 2020. "Bibliometrics-based decision tree (BBDT) for deciding whether two universities in the Leiden ranking differ substantially in their performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 1255-1258, February.
    17. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    18. Heidi Peterson, 2023. "Cost–Benefit Analysis (CBA) or the Highway? An Alternative Road to Investigating the Value for Money of International Development Research," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 35(2), pages 260-280, April.
    19. Alba Viana Lora & Marta Gemma Nel-lo Andreu, 2020. "Alternative Metrics for Assessing the Social Impact of Tourism Research," Sustainability, MDPI, vol. 12(10), pages 1-12, May.
    20. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:scippl:v:49:y:2022:i:2:p:313-323.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/spp .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.