IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v49y2020i4s0048733320300330.html
   My bibliography  Save this article

Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰

Author

Listed:
  • Ginther, Donna K.
  • Heggeness, Misty L.

Abstract

The scientific community is engaged in an active debate on the value of its peer-review system. Does peer review actually serve the role we envision for it—that of helping government agencies predict what ideas have the best chance of contributing to scientific advancement? Many federal agencies use a two-step review process that includes programmatic discretion in selecting awards. This process allows us to determine whether success in a future independent scientific-research career is more accurately predicted by peer-review recommendations or discretion by program staff and institute leaders. Using data from a prestigious training program at the National Institute of Health (NIH), the Ruth L. Kirschstein National Research Service Award (NRSA), we provide evidence on the efficacy of peer review. We find that, despite all current claims to the contrary, the existing peer-review system works as intended. It more closely predicts high-quality science and future research independence than discretion. We discover also that regression discontinuity, the econometric method typically used to examine the effect of scientific funding, does not fit many scientific-funding models and should only be used with caution when studying federal awards for science.

Suggested Citation

  • Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).
    • Handle: RePEc:eee:respol:v:49:y:2020:i:4:s0048733320300330
    DOI: 10.1016/j.respol.2020.103953
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733320300330
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2020.103953?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mitchell Hoffman & Lisa B Kahn & Danielle Li, 2018. "Discretion in Hiring," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(2), pages 765-800.
    2. Benavente, José Miguel & Crespi, Gustavo & Figal Garone, Lucas & Maffioli, Alessandro, 2012. "The impact of national research funds: A regression discontinuity approach to the Chilean FONDECYT," Research Policy, Elsevier, vol. 41(8), pages 1461-1475.
    3. Pierre Azoulay & Joshua S Graff Zivin & Danielle Li & Bhaven N Sampat, 2019. "Public R&D Investments and Private-sector Patenting: Evidence from NIH Funding Rules," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 86(1), pages 117-152.
    4. Matthew Blackwell & Stefano Iacus & Gary King & Giuseppe Porro, 2009. "cem: Coarsened exact matching in Stata," Stata Journal, StataCorp LP, vol. 9(4), pages 524-546, December.
    5. Stephen A Gallo & Joanne H Sullivan & Scott R Glisson, 2016. "The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications," PLOS ONE, Public Library of Science, vol. 11(10), pages 1-18, October.
    6. Ayoubi, Charles & Pezzoni, Michele & Visentin, Fabiana, 2019. "The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions?," Research Policy, Elsevier, vol. 48(1), pages 84-97.
    7. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of research grant funding on scientific productivity," Journal of Public Economics, Elsevier, vol. 95(9), pages 1168-1177.
    8. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of NIH postdoctoral training grants on scientific productivity," Research Policy, Elsevier, vol. 40(6), pages 864-874, July.
    9. Sabrina T. Howell, 2017. "Financing Innovation: Evidence from R&D Grants," American Economic Review, American Economic Association, vol. 107(4), pages 1136-1164, April.
    10. Guido W. Imbens, 2015. "Matching Methods in Practice: Three Examples," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 373-419.
    11. Thijs Bol & Mathijs de Vaan & Arnout van de Rijt, 2018. "The Matthew effect in science funding," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(19), pages 4887-4890, May.
    12. Misty L. Heggeness & Donna K. Ginther & Maria I. Larenas & Frances D. Carter-Johnson, 2018. "The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research," NBER Working Papers 24508, National Bureau of Economic Research, Inc.
    13. Elizabeth L. Pier & Markus Brauer & Amarette Filut & Anna Kaatz & Joshua Raclaw & Mitchell J. Nathan & Cecilia E. Ford & Molly Carnes, 2018. "Low agreement among reviewers evaluating the same NIH grant applications," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(12), pages 2952-2957, March.
    14. Iacus, Stefano & King, Gary & Porro, Giuseppe, 2009. "cem: Software for Coarsened Exact Matching," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 30(i09).
    15. Stephen A Gallo & Afton S Carpenter & David Irwin & Caitlin D McPartland & Joseph Travis & Sofie Reynders & Lisa A Thompson & Scott R Glisson, 2014. "The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-9, September.
    16. Danielle Li, 2017. "Expertise versus Bias in Evaluation: Evidence from the NIH," American Economic Journal: Applied Economics, American Economic Association, vol. 9(2), pages 60-92, April.
    17. Jon Kleinberg & Himabindu Lakkaraju & Jure Leskovec & Jens Ludwig & Sendhil Mullainathan, 2018. "Human Decisions and Machine Predictions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(1), pages 237-293.
    18. Luca Grilli & Samuele Murtinu, 2011. "Econometric Evaluation of Public Policies for Science and Innovation: A Brief Guide to Practice," Chapters, in: Massimo G. Colombo & Luca Grilli & Lucia Piscitello & Cristina Rossi-Lamastra (ed.), Science and Innovation Policy for the New Knowledge Economy, chapter 3, Edward Elgar Publishing.
    19. King, Gary & Nielsen, Richard, 2019. "Why Propensity Scores Should Not Be Used for Matching," Political Analysis, Cambridge University Press, vol. 27(4), pages 435-454, October.
    20. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2013. "National Institutes of Health Peer Review: Challenges and Avenues for Reform," Innovation Policy and the Economy, University of Chicago Press, vol. 13(1), pages 1-22.
    21. Massimo G. Colombo & Luca Grilli & Lucia Piscitello & Cristina Rossi-Lamastra (ed.), 2011. "Science and Innovation Policy for the New Knowledge Economy," Books, Edward Elgar Publishing, number 13746.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    2. Bol, Thijs & de Vaan, Mathijs & van de Rijt, Arnout, 2022. "Gender-equal funding rates conceal unequal evaluations," Research Policy, Elsevier, vol. 51(1).
    3. Azoulay, Pierre & Greenblatt, Wesley H. & Heggeness, Misty L., 2021. "Long-term effects from early exposure to research: Evidence from the NIH “Yellow Berets”," Research Policy, Elsevier, vol. 50(9).
    4. Cruz-Castro, Laura & Sanz-Menendez, Luis, 2021. "What should be rewarded? Gender and evaluation criteria for tenure and promotion," Journal of Informetrics, Elsevier, vol. 15(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Donna K. Ginther & Misty L. Heggeness, 2020. "Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program," NBER Working Papers 26841, National Bureau of Economic Research, Inc.
    2. Misty L. Heggeness & Donna K. Ginther & Maria I. Larenas & Frances D. Carter-Johnson, 2018. "The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research," NBER Working Papers 24508, National Bureau of Economic Research, Inc.
    3. Lawson, Cornelia & Salter, Ammon, 2023. "Exploring the effect of overlapping institutional applications on panel decision-making," Research Policy, Elsevier, vol. 52(9).
    4. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    5. Marielle Non & Jeroen van Honk & Vince van Houten & Inge van der Weijden & Thed van Leeuwen, 2022. "Getting off to a flying start? The effects of an early-career international mobility grant on scientific performance," CPB Discussion Paper 443, CPB Netherlands Bureau for Economic Policy Analysis.
    6. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Working Papers 26889, National Bureau of Economic Research, Inc.
    7. Graddy-Reed, Alexandra & Lanahan, Lauren & D'Agostino, Jesse, 2021. "Training across the academy: The impact of R&D funding on graduate students," Research Policy, Elsevier, vol. 50(5).
    8. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Chapters, in: Innovation and Public Policy, pages 117-150, National Bureau of Economic Research, Inc.
    9. Lawson, Cornelia & Geuna, Aldo & Finardi, Ugo, 2021. "The funding-productivity-gender nexus in science, a multistage analysis," Research Policy, Elsevier, vol. 50(3).
    10. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    11. Ryazanova, Olga & Jaskiene, Jolanta, 2022. "Managing individual research productivity in academic organizations: A review of the evidence and a path forward," Research Policy, Elsevier, vol. 51(2).
    12. Graddy-Reed, Alexandra & Lanahan, Lauren & Eyer, Jonathan, 2019. "Gender discrepancies in publication productivity of high-performing life science graduate students," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    13. Philipp vom Berge & Achim Schmillen, 2023. "Effects of mass layoffs on local employment—evidence from geo-referenced data," Journal of International Economic Law, Oxford University Press, vol. 23(3), pages 509-539.
    14. Matteo Aquilina & Giulio Cornelli & Marina Sanchez del Villar, 2024. "Regulation, information asymmetries and the funding of new ventures," BIS Working Papers 1162, Bank for International Settlements.
    15. Lauren Lanahan & Daniel Armanios, 2018. "Does More Certification Always Benefit a Venture?," Organization Science, INFORMS, vol. 29(5), pages 931-947, October.
    16. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    17. Ravi Bapna & Alok Gupta & Gautam Ray & Shweta Singh, 2023. "Single-Sourcing vs. Multisourcing: An Empirical Analysis of Large Information Technology Outsourcing Arrangements," Information Systems Research, INFORMS, vol. 34(3), pages 1109-1130, September.
    18. Maria Abreu & Özge Öner, 2020. "Disentangling the Brexit vote: The role of economic, social and cultural contexts in explaining the UK’s EU referendum vote," Environment and Planning A, , vol. 52(7), pages 1434-1456, October.
    19. Katrin Hussinger & João N. Carvalho, 2022. "The long-term effect of research grants on the scientific output of university professors," Industry and Innovation, Taylor & Francis Journals, vol. 29(4), pages 463-487, April.
    20. Kyle R. Myers, 2022. "Some Tradeoffs of Competition in Grant Contests," Papers 2207.02379, arXiv.org, revised Mar 2024.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:49:y:2020:i:4:s0048733320300330. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.