IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/26841.html
   My bibliography  Save this paper

Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program

Author

Listed:
  • Donna K. Ginther
  • Misty L. Heggeness

Abstract

The scientific community is engaged in an active debate on the value of its peer-review system. Does peer review actually serve the role we envision for it—that of helping government agencies predict what ideas have the best chance of contributing to scientific advancement? Many federal agencies use a two-step review process that includes programmatic discretion in selecting awards. This process allows us to determine whether success in a future independent scientific-research career is more accurately predicted by peer-review recommendations or discretion by program staff and institute leaders. Using data from a prestigious training program at the National Institute of Health (NIH), the Ruth L. Kirschstein National Research Service Award (NRSA), we provide evidence on the efficacy of peer review. We find that, despite all current claims to the contrary, the existing peer-review system works as intended. It more closely predicts high-quality science and future research independence than discretion. We discover also that regression discontinuity, the econometric method typically used to examine the effect of scientific funding, does not fit many scientific-funding models and should only be used with caution when studying federal awards for science.

Suggested Citation

  • Donna K. Ginther & Misty L. Heggeness, 2020. "Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program," NBER Working Papers 26841, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:26841
    Note: LS PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w26841.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Benavente, José Miguel & Crespi, Gustavo & Figal Garone, Lucas & Maffioli, Alessandro, 2012. "The impact of national research funds: A regression discontinuity approach to the Chilean FONDECYT," Research Policy, Elsevier, vol. 41(8), pages 1461-1475.
    2. Pierre Azoulay & Joshua S Graff Zivin & Danielle Li & Bhaven N Sampat, 2019. "Public R&D Investments and Private-sector Patenting: Evidence from NIH Funding Rules," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 86(1), pages 117-152.
    3. Stephen A Gallo & Joanne H Sullivan & Scott R Glisson, 2016. "The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications," PLOS ONE, Public Library of Science, vol. 11(10), pages 1-18, October.
    4. Ayoubi, Charles & Pezzoni, Michele & Visentin, Fabiana, 2019. "The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions?," Research Policy, Elsevier, vol. 48(1), pages 84-97.
    5. Sabrina T. Howell, 2017. "Financing Innovation: Evidence from R&D Grants," American Economic Review, American Economic Association, vol. 107(4), pages 1136-1164, April.
    6. Guido W. Imbens, 2015. "Matching Methods in Practice: Three Examples," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 373-419.
    7. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    8. Misty L. Heggeness & Donna K. Ginther & Maria I. Larenas & Frances D. Carter-Johnson, 2018. "The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research," NBER Working Papers 24508, National Bureau of Economic Research, Inc.
    9. Elizabeth L. Pier & Markus Brauer & Amarette Filut & Anna Kaatz & Joshua Raclaw & Mitchell J. Nathan & Cecilia E. Ford & Molly Carnes, 2018. "Low agreement among reviewers evaluating the same NIH grant applications," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(12), pages 2952-2957, March.
    10. Stephen A Gallo & Afton S Carpenter & David Irwin & Caitlin D McPartland & Joseph Travis & Sofie Reynders & Lisa A Thompson & Scott R Glisson, 2014. "The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-9, September.
    11. Danielle Li, 2017. "Expertise versus Bias in Evaluation: Evidence from the NIH," American Economic Journal: Applied Economics, American Economic Association, vol. 9(2), pages 60-92, April.
    12. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2013. "National Institutes of Health Peer Review: Challenges and Avenues for Reform," Innovation Policy and the Economy, University of Chicago Press, vol. 13(1), pages 1-22.
    13. Massimo G. Colombo & Luca Grilli & Lucia Piscitello & Cristina Rossi-Lamastra (ed.), 2011. "Science and Innovation Policy for the New Knowledge Economy," Books, Edward Elgar Publishing, number 13746.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    2. Lawson, Cornelia & Salter, Ammon, 2023. "Exploring the effect of overlapping institutional applications on panel decision-making," Research Policy, Elsevier, vol. 52(9).
    3. Bol, Thijs & de Vaan, Mathijs & van de Rijt, Arnout, 2022. "Gender-equal funding rates conceal unequal evaluations," Research Policy, Elsevier, vol. 51(1).
    4. Azoulay, Pierre & Greenblatt, Wesley H. & Heggeness, Misty L., 2021. "Long-term effects from early exposure to research: Evidence from the NIH “Yellow Berets”," Research Policy, Elsevier, vol. 50(9).
    5. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Working Papers 26889, National Bureau of Economic Research, Inc.
    6. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Chapters, in: Innovation and Public Policy, pages 117-150, National Bureau of Economic Research, Inc.
    7. Cruz-Castro, Laura & Sanz-Menendez, Luis, 2021. "What should be rewarded? Gender and evaluation criteria for tenure and promotion," Journal of Informetrics, Elsevier, vol. 15(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).
    2. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Chapters, in: Innovation and Public Policy, pages 117-150, National Bureau of Economic Research, Inc.
    3. Pierre Azoulay & Danielle Li, 2020. "Scientific Grant Funding," NBER Working Papers 26889, National Bureau of Economic Research, Inc.
    4. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    5. Margaret K. Kyle, 2019. "The Alignment of Innovation Policy and Social Welfare: Evidence from Pharmaceuticals," NBER Chapters, in: Innovation Policy and the Economy, Volume 20, pages 95-123, National Bureau of Economic Research, Inc.
    6. Hünermund, Paul & Czarnitzki, Dirk, 2019. "Estimating the causal effect of R&D subsidies in a pan-European program," Research Policy, Elsevier, vol. 48(1), pages 115-124.
    7. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    8. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    9. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    10. Lawson, Cornelia & Salter, Ammon, 2023. "Exploring the effect of overlapping institutional applications on panel decision-making," Research Policy, Elsevier, vol. 52(9).
    11. Kyle R. Myers, 2022. "Some Tradeoffs of Competition in Grant Contests," Papers 2207.02379, arXiv.org, revised Mar 2024.
    12. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," CESifo Working Paper Series 7268, CESifo.
    13. Amitabh Chandra & Courtney Coile & Corina Mommaerts, 2023. "What Can Economics Say about Alzheimer's Disease?," Journal of Economic Literature, American Economic Association, vol. 61(2), pages 428-470, June.
    14. Wen Ci & Jose Galdo & Marcel Voia & Christopher Worswick, 2015. "Wage returns to mid-career investments in job training through employer supported course enrollment: evidence for Canada," IZA Journal of Labor Policy, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-25, December.
    15. Huseyin Emre Sayici & Mehmet Fatih Ulu, 2023. "Economic Effects of R&D Supports," Koç University-TUSIAD Economic Research Forum Working Papers 2308, Koc University-TUSIAD Economic Research Forum.
    16. Eva-Maria Egger & Julie Litchfield, 2019. "Following in their footsteps: an analysis of the impact of successive migration on rural household welfare in Ghana," IZA Journal of Migration and Development, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 9(1), pages 1-22, December.
    17. Wagener, Andreas & Zenker, Juliane, 2015. "Stochastic Transfers, Risky Investment and Incomes: Evidence from an Income Guarantee Program in Thailand," Hannover Economic Papers (HEP) dp-562, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät.
    18. Goller, Daniel & Lechner, Michael & Moczall, Andreas & Wolff, Joachim, 2020. "Does the estimation of the propensity score by machine learning improve matching estimation? The case of Germany's programmes for long term unemployed," Labour Economics, Elsevier, vol. 65(C).
    19. Lee, Ji Yong & Nayga Jr, Rodolfo M. & Jo, Young & Restrepo, Brandon J., 2022. "Time use and eating patterns of SNAP participants over the benefit month," Food Policy, Elsevier, vol. 106(C).
    20. Dirk Czarnitzki & Paul Hünermund & Nima Moshgbar, 2018. "Public procurement as policy instrument for innovation," Working Papers of Department of Management, Strategy and Innovation, Leuven 606259, KU Leuven, Faculty of Economics and Business (FEB), Department of Management, Strategy and Innovation, Leuven.

    More about this item

    JEL classification:

    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity
    • O3 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights
    • O38 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Government Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:26841. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.