IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/24358.html
   My bibliography  Save this paper

Learning When to Quit: An Empirical Model of Experimentation

Author

Listed:
  • Bernhard Ganglmair
  • Timothy Simcoe
  • Emanuele Tarantino

Abstract

Research productivity depends on the ability to discern whether an idea is promising, and a willingness to abandon the ones that are not. Economists know little about this process, however, because empirical studies of innovation typically begin with a sample of issued patents or published papers that were already selected from a pool of promising ideas. This paper unpacks the idea selection process using a unique dataset from the Internet Engineering Task Force (IETF), a voluntary organization that develops protocols for managing Internet infrastructure. For a large sample of IETF proposals, we observe a sequence of decisions to either revise, publish, or abandon the underlying idea, along with changes to the proposal and the demographics of the author team. Using these data, we provide a descriptive analysis of how R&D is conducted within the IETF, and estimate a dynamic discrete choice model whose key parameters measure the speed at which author teams learn whether they have a good (i.e., publishable) idea. The estimates imply that sixty percent of IETF proposals are publishable, but only one-third of the good ideas survive the review process. Author experience and increased attention from the IETF community are associated with faster learning. Finally, we simulate two counterfactual innovation policies: an R&D subsidy and a publication-prize. Subsidies have a larger impact on research output, though prizes perform better when accounting for researchers' opportunity costs.

Suggested Citation

  • Bernhard Ganglmair & Timothy Simcoe & Emanuele Tarantino, 2018. "Learning When to Quit: An Empirical Model of Experimentation," NBER Working Papers 24358, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:24358
    Note: PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w24358.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jesús Fernández-Villaverde & Juan F. Rubio-Ramírez, 2007. "Estimating Macroeconomic Models: A Likelihood Approach," Review of Economic Studies, Oxford University Press, vol. 74(4), pages 1059-1087.
    2. Roberts, Kevin & Weitzman, Martin L, 1981. "Funding Criteria for Research, Development, and Exploration Projects," Econometrica, Econometric Society, vol. 49(5), pages 1261-1288, September.
    3. Nicholas Bloom & Luis Garicano & Raffaella Sadun & John Van Reenen, 2014. "The Distinct Effects of Information Technology and Communication Technology on Firm Organization," Management Science, INFORMS, vol. 60(12), pages 2859-2885, December.
    4. Timothy Simcoe, 2012. "Standard Setting Committees: Consensus Governance for Shared Technology Platforms," American Economic Review, American Economic Association, vol. 102(1), pages 305-336, February.
    5. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2011. "Incentives and creativity: evidence from the academic life sciences," RAND Journal of Economics, RAND Corporation, vol. 42(3), pages 527-554, September.
    6. Thierry Magnac & David Thesmar, 2002. "Identifying Dynamic Discrete Decision Processes," Econometrica, Econometric Society, vol. 70(2), pages 801-816, March.
    7. Godfrey Keller & Sven Rady & Martin Cripps, 2005. "Strategic Experimentation with Exponential Bandits," Econometrica, Econometric Society, vol. 73(1), pages 39-68, January.
    8. Patrick Bolton & Christopher Harris, 1999. "Strategic Experimentation," Econometrica, Econometric Society, vol. 67(2), pages 349-374, March.
    9. Galasso, Alberto & Mitchell, Matthew & Virag, Gabor, 2016. "Market outcomes and dynamic patent buyouts," International Journal of Industrial Organization, Elsevier, vol. 48(C), pages 207-243.
    10. David Hirshleifer & Angie Low & Siew Hong Teoh, 2012. "Are Overconfident CEOs Better Innovators?," Journal of Finance, American Finance Association, vol. 67(4), pages 1457-1498, August.
    11. Tat Y. Chan & Barton H. Hamilton, 2006. "Learning, Private Information, and the Economic Evaluation of Randomized Experiments," Journal of Political Economy, University of Chicago Press, vol. 114(6), pages 997-1040, December.
    12. Nicolas Serrano-Velarde & Douglas Hanley & Ufuk Akcigit, 2012. "Back to Basics: Basic Research Spillovers, Innovation Policy and Growth," 2012 Meeting Papers 665, Society for Economic Dynamics.
    13. Hugo Hopenhayn & Gerard Llobet & Matthew Mitchell, 2006. "Rewarding Sequential Innovators: Prizes, Patents, and Buyouts," Journal of Political Economy, University of Chicago Press, vol. 114(6), pages 1041-1068, December.
    14. Richard Ericson & Ariel Pakes, 1995. "Markov-Perfect Industry Dynamics: A Framework for Empirical Work," Review of Economic Studies, Oxford University Press, vol. 62(1), pages 53-82.
    15. Heidhues, Paul & Rady, Sven & Strack, Philipp, 2015. "Strategic experimentation with private payoffs," Journal of Economic Theory, Elsevier, vol. 159(PA), pages 531-551.
    16. Richard Gilbert & Carl Shapiro, 1990. "Optimal Patent Length and Breadth," RAND Journal of Economics, The RAND Corporation, vol. 21(1), pages 106-112, Spring.
    17. Pakes, Ariel S, 1986. "Patents as Options: Some Estimates of the Value of Holding European Patent Stocks," Econometrica, Econometric Society, vol. 54(4), pages 755-784, July.
    18. Giuseppe Moscarini & Lones Smith, 2001. "The Optimal Level of Experimentation," Econometrica, Econometric Society, vol. 69(6), pages 1629-1644, November.
    19. Sabrina T. Howell, 2017. "Reducing Information Frictions in Venture Capital: The Role of New Venture Competitions," NBER Working Papers 23874, National Bureau of Economic Research, Inc.
    20. William R. Kerr & Ramana Nanda & Matthew Rhodes-Kropf, 2014. "Entrepreneurship as Experimentation," Journal of Economic Perspectives, American Economic Association, vol. 28(3), pages 25-48, Summer.
    21. Bernhard Ganglmair & Emanuele Tarantino, 2014. "Conversation with secrets," RAND Journal of Economics, RAND Corporation, vol. 45(2), pages 273-302, June.
    22. Michael Roach & Wesley M. Cohen, 2013. "Lens or Prism? Patent Citations as a Measure of Knowledge Flows from Public Research," Management Science, INFORMS, vol. 59(2), pages 504-525, October.
    23. Weitzman, Martin L, 1979. "Optimal Search for the Best Alternative," Econometrica, Econometric Society, vol. 47(3), pages 641-654, May.
    24. Barton H. Hamilton & Jack A. Nickerson & Hideo Owan, 2003. "Team Incentives and Worker Heterogeneity: An Empirical Analysis of the Impact of Teams on Productivity and Participation," Journal of Political Economy, University of Chicago Press, vol. 111(3), pages 465-497, June.
    25. Andrew T. Ching & Tülin Erdem & Michael P. Keane, 2013. "Invited Paper ---Learning Models: An Assessment of Progress, Challenges, and New Developments," Marketing Science, INFORMS, vol. 32(6), pages 913-938, November.
    26. Tülin Erdem & Michael P. Keane, 1996. "Decision-Making Under Uncertainty: Capturing Dynamic Brand Choice Processes in Turbulent Consumer Goods Markets," Marketing Science, INFORMS, vol. 15(1), pages 1-20.
    27. Moscarini, Giuseppe & Squintani, Francesco, 2010. "Competitive experimentation with private information: The survivor's curse," Journal of Economic Theory, Elsevier, vol. 145(2), pages 639-660, March.
    28. Murray, Fiona & Stern, Scott & Campbell, Georgina & MacCormack, Alan, 2012. "Grand Innovation Prizes: A theoretical, normative, and empirical evaluation," Research Policy, Elsevier, vol. 41(10), pages 1779-1792.
    29. Gregory S. Crawford & Matthew Shum, 2005. "Uncertainty and Learning in Pharmaceutical Demand," Econometrica, Econometric Society, vol. 73(4), pages 1137-1173, July.
    30. Robert L. Bray & Decio Coviello & Andrea Ichino & Nicola Persico, 2016. "Multitasking, Multiarmed Bandits, and the Italian Judiciary," Manufacturing & Service Operations Management, INFORMS, vol. 18(4), pages 545-558, October.
    31. repec:oup:jeurec:v:14:y:2016:i:4:p:828-870. is not listed on IDEAS
    32. Glenn Ellison, 2002. "Evolving Standards for Academic Publishing: A q-r Theory," Journal of Political Economy, University of Chicago Press, vol. 110(5), pages 994-1034, October.
    33. repec:bla:randje:v:48:y:2017:i:2:p:438-466 is not listed on IDEAS
    34. Alberto Galasso & Timothy S. Simcoe, 2011. "CEO Overconfidence and Innovation," Management Science, INFORMS, vol. 57(8), pages 1469-1484, August.
    35. Andrew T. Ching & Tülin Erdem & Michael P. Keane, 2013. "Learning Models: An Assessment of Progress, Challenges and New Developments," Economics Papers 2013-W07, Economics Group, Nuffield College, University of Oxford.
    36. Ufuk Akcigit & Qingmin Liu, 2016. "The Role of Information in Innovation and Competition," Journal of the European Economic Association, European Economic Association, vol. 14(4), pages 828-870.
    37. Daniel P. Gross, 2017. "Performance feedback in competitive product development," RAND Journal of Economics, RAND Corporation, vol. 48(2), pages 438-466, May.
    Full references (including those not matched with items on IDEAS)

    More about this item

    JEL classification:

    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • O31 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Innovation and Invention: Processes and Incentives
    • O32 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Management of Technological Innovation and R&D

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:24358. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: http://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.