IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/24541.html

Finding Needles in Haystacks: Artificial Intelligence and Recombinant Growth

Author

Listed:
  • Ajay Agrawal
  • John McHale
  • Alex Oettl

Abstract

Innovation is often predicated on discovering useful new combinations of existing knowledge in highly complex knowledge spaces. These needle-in-a-haystack type problems are pervasive in fields like genomics, drug discovery, materials science, and particle physics. We develop a combinatorial-based knowledge production function and embed it in the classic Jones growth model (1995) to explore how breakthroughs in artificial intelligence (AI) that dramatically improve prediction accuracy about which combinations have the highest potential could enhance discovery rates and consequently economic growth. This production function is a generalization (and reinterpretation) of the Romer/Jones knowledge production function. Separate parameters control the extent of individual-researcher knowledge access, the effects of fishing out/complexity, and the ease of forming research teams.

Suggested Citation

  • Ajay Agrawal & John McHale & Alex Oettl, 2018. "Finding Needles in Haystacks: Artificial Intelligence and Recombinant Growth," NBER Working Papers 24541, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:24541
    Note: EFG PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w24541.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Robert J. Gordon, 2016. "The Rise and Fall of American Growth: The U.S. Standard of Living since the Civil War," Economics Books, Princeton University Press, edition 1, number 10544, December.
    2. Ajay Agrawal & Avi Goldfarb, 2008. "Restructuring Research: Communication Costs and the Democratization of University Innovation," American Economic Review, American Economic Association, vol. 98(4), pages 1578-1590, September.
    3. Nicholas Bloom & Charles I. Jones & John Van Reenen & Michael Webb, 2020. "Are Ideas Getting Harder to Find?," American Economic Review, American Economic Association, vol. 110(4), pages 1104-1144, April.
    4. Ajay Agrawal & Joshua Gans & Avi Goldfarb, 2018. "Prediction, Judgment, and Complexity: A Theory of Decision-Making and Artificial Intelligence," NBER Chapters, in: The Economics of Artificial Intelligence: An Agenda, pages 89-110, National Bureau of Economic Research, Inc.
    5. Jones, Charles I., 2005. "Growth and Ideas," Handbook of Economic Growth, in: Philippe Aghion & Steven Durlauf (ed.), Handbook of Economic Growth, edition 1, volume 1, chapter 16, pages 1063-1111, Elsevier.
    6. Jeffrey L. Furman & Scott Stern, 2011. "Climbing atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research," American Economic Review, American Economic Association, vol. 101(5), pages 1933-1963, August.
    7. Philippe Aghion & Benjamin F. Jones & Charles I. Jones, 2018. "Artificial Intelligence and Economic Growth," NBER Chapters, in: The Economics of Artificial Intelligence: An Agenda, pages 237-282, National Bureau of Economic Research, Inc.
    8. Sendhil Mullainathan & Jann Spiess, 2017. "Machine Learning: An Applied Econometric Approach," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 87-106, Spring.
    9. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    10. Bresnahan, Timothy F. & Trajtenberg, M., 1995. "General purpose technologies 'Engines of growth'?," Journal of Econometrics, Elsevier, vol. 65(1), pages 83-108, January.
    11. Benjamin F. Jones, 2009. "The Burden of Knowledge and the "Death of the Renaissance Man": Is Innovation Getting Harder?," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(1), pages 283-317.
    12. Jones, Charles I, 1995. "R&D-Based Models of Economic Growth," Journal of Political Economy, University of Chicago Press, vol. 103(4), pages 759-784, August.
    13. Lee Fleming & Olav Sorenson, 2004. "Science as a map in technological search," Strategic Management Journal, Wiley Blackwell, vol. 25(8‐9), pages 909-928, August.
    14. Adam B. Jaffe & Manuel Trajtenberg & Rebecca Henderson, 1993. "Geographic Localization of Knowledge Spillovers as Evidenced by Patent Citations," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 108(3), pages 577-598.
    15. Agrawal, Ajay & Kapur, Devesh & McHale, John, 2008. "How do spatial and social proximity influence knowledge flows? Evidence from patent data," Journal of Urban Economics, Elsevier, vol. 64(2), pages 258-269, September.
    16. Iain M. Cockburn & Rebecca Henderson & Scott Stern, 2018. "The Impact of Artificial Intelligence on Innovation," NBER Working Papers 24449, National Bureau of Economic Research, Inc.
    17. William D. Nordhaus, 2021. "Are We Approaching an Economic Singularity? Information Technology and the Future of Economic Growth," American Economic Journal: Macroeconomics, American Economic Association, vol. 13(1), pages 299-332, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. J. Klinger & J. Mateos-Garcia & K. Stathoulopoulos, 2018. "Deep learning, deep change? Mapping the development of the Artificial Intelligence General Purpose Technology," Papers 1808.06355, arXiv.org.
    2. Bianchini, Stefano & Müller, Moritz & Pelletier, Pierre, 2022. "Artificial intelligence in science: An emerging general method of invention," Research Policy, Elsevier, vol. 51(10).
    3. Gries, Thomas & Naudé, Wim, 2018. "Artificial Intelligence, Jobs, Inequality and Productivity: Does Aggregate Demand Matter?," IZA Discussion Papers 12005, IZA Network @ LISER.
    4. Hu, Xiaohua & Fei, Yulian, 2023. "Does the matching degree between data and human capital affect firm innovation?," Finance Research Letters, Elsevier, vol. 55(PB).
    5. Andrea Borsato & Patrick Llerena, 2026. "The US university-industry link in the R&D of AI: Back to the origins?," Journal of Evolutionary Economics, Springer, vol. 36(1), pages 1-39, April.
    6. Erik Brynjolfsson & Daniel Rock & Chad Syverson, 2021. "The Productivity J-Curve: How Intangibles Complement General Purpose Technologies," American Economic Journal: Macroeconomics, American Economic Association, vol. 13(1), pages 333-372, January.
    7. Stefano Bianchini & Moritz Müller & Pierre Pelletier, 2022. "Artificial intelligence in science: An emerging general method of invention," Post-Print hal-03958025, HAL.
    8. Ayoubi, Charles, 2020. "Machine learning in healthcare: Mirage or miracle for breaking the costs dead-lock?," Thesis Commons tc24d, Center for Open Science.
    9. Seth G. Benzell & Erik Brynjolfsson, 2019. "Digital Abundance and Scarce Genius: Implications for Wages, Interest Rates, and Growth," NBER Working Papers 25585, National Bureau of Economic Research, Inc.
    10. Majid Majzoubi & Eric Yanfei Zhao, 2023. "Going beyond optimal distinctiveness: Strategic positioning for gaining an audience composition premium," Strategic Management Journal, Wiley Blackwell, vol. 44(3), pages 737-777, March.
    11. Naudé, Wim, 2022. "The Future Economics of Artificial Intelligence: Mythical Agents, a Singleton and the Dark Forest," IZA Discussion Papers 15713, IZA Network @ LISER.
    12. Joel Klinger & Juan Mateos-Garcia & Konstantinos Stathoulopoulos, 2021. "Deep learning, deep change? Mapping the evolution and geography of a general purpose technology," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5589-5621, July.
    13. Gersbach, Hans & Komarov, Evgenij & von Maydell, Richard, 2025. "Artificial intelligence as self-learning capital," Economic Modelling, Elsevier, vol. 153(C).
    14. Naudé, Wim, 2023. "Melancholy Hues: The Futility of Green Growth and Degrowth, and the Inevitability of Societal Collapse," IZA Discussion Papers 16139, IZA Network @ LISER.
    15. Georgios A. Tritsaris, 2025. "Occupational Tasks, Automation, and Economic Growth: A Modeling and Simulation Approach," Papers 2512.16261, arXiv.org, revised Dec 2025.
    16. Li, Dongkun & Chen, Yufeng & Miao, Jiafeng, 2022. "Does ICT create a new driving force for manufacturing?—Evidence from Chinese manufacturing firms," Telecommunications Policy, Elsevier, vol. 46(1).
    17. Carvalho, V. M. & Freund, L. B., 2026. "Firms as Foragers," Cambridge Working Papers in Economics 2659, Faculty of Economics, University of Cambridge.
    18. Wu, Yifan & Yuan, Yiming & Song, Xueyin, 2025. "The impact of AI adoption on R&D productivity: Evidence from Chinese pharmaceutical manufacturing industry," Journal of Asian Economics, Elsevier, vol. 97(C).
    19. Bianchini, Stefano & Müller, Moritz & Pelletier, Pierre, 2025. "Drivers and barriers of AI adoption and use in scientific research," Technological Forecasting and Social Change, Elsevier, vol. 220(C).
    20. Chen, Zhongfei & Jiang, Kangqi, 2024. "Digitalization and corporate investment efficiency: Evidence from China," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 91(C).
    21. Bianchini, Stefano & Di Girolamo, Valentina & Ravet, Julien & Arranz, David, 2026. "AI in science: When and where it makes a difference," Research Policy, Elsevier, vol. 55(6).
    22. Simone Vannuccini & Ekaterina Prytkova, 2021. "Artificial Intelligence’s New Clothes? From General Purpose Technology to Large Technical System," SPRU Working Paper Series 2021-02, SPRU - Science Policy Research Unit, University of Sussex Business School.
    23. Li, Hao & Wang, Gaowang & Yang, Liyang, 2024. "Data-driven innovation and growth," MPRA Paper 122388, University Library of Munich, Germany.
    24. Lan, Mudan & Huang, Yuhong, 2026. "Can industrial technology convergence facilitate firms' international expansion? Firm-level empirical evidence from China," Economic Modelling, Elsevier, vol. 158(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wu, Yifan & Yuan, Yiming & Song, Xueyin, 2025. "The impact of AI adoption on R&D productivity: Evidence from Chinese pharmaceutical manufacturing industry," Journal of Asian Economics, Elsevier, vol. 97(C).
    2. Stefano Bianchini & Moritz Muller & Pierre Pelletier, 2020. "Deep Learning in Science," Papers 2009.01575, arXiv.org, revised Sep 2020.
    3. Ina Ganguli & Jeffrey Lin & Vitaly Meursault & Nicholas Reynolds, 2026. "Spreading Out Across Expanding Idea Space," Working Papers 26-39, Federal Reserve Bank of Philadelphia.
    4. Choi, Jin-Uk & Lee, Chang-Yang, 2022. "The differential effects of basic research on firm R&D productivity: The conditioning role of technological diversification," Technovation, Elsevier, vol. 118(C).
    5. Bianchini, Stefano & Di Girolamo, Valentina & Ravet, Julien & Arranz, David, 2026. "AI in science: When and where it makes a difference," Research Policy, Elsevier, vol. 55(6).
    6. Venturini, Francesco, 2022. "Intelligent technologies and productivity spillovers: Evidence from the Fourth Industrial Revolution," Journal of Economic Behavior & Organization, Elsevier, vol. 194(C), pages 220-243.
    7. Ajay Agrawal & Joshua Gans & Avi Goldfarb, 2019. "Economic Policy for Artificial Intelligence," Innovation Policy and the Economy, University of Chicago Press, vol. 19(1), pages 139-159.
    8. Philippe Aghion & Benjamin F. Jones & Charles I. Jones, 2018. "Artificial Intelligence and Economic Growth," NBER Chapters, in: The Economics of Artificial Intelligence: An Agenda, pages 237-282, National Bureau of Economic Research, Inc.
    9. Parteka, Aleksandra & Kordalska, Aleksandra, 2023. "Artificial intelligence and productivity: global evidence from AI patent and bibliometric data," Technovation, Elsevier, vol. 125(C).
    10. Callaghan, Christian William, 2021. "Growth contributions of technological change: Is there a burden of knowledge effect?," Technological Forecasting and Social Change, Elsevier, vol. 172(C).
    11. Jorge Guzman & Fiona Murray & Scott Stern & Heidi Williams, 2024. "Accelerating Innovation Ecosystems: The Promise and Challenges of Regional Innovation Engines," Entrepreneurship and Innovation Policy and the Economy, University of Chicago Press, vol. 3(1), pages 9-75.
    12. Ian Goldin & Pantelis Koutroumpis & François Lafond & Julian Winkler, 2024. "Why Is Productivity Slowing Down?," Journal of Economic Literature, American Economic Association, vol. 62(1), pages 196-268, March.
    13. Gries, Thomas & Naude, Wim, 2018. "Artificial intelligence, jobs, inequality and productivity: Does aggregate demand matter?," MERIT Working Papers 2018-047, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    14. Bianchini, Stefano & Müller, Moritz & Pelletier, Pierre, 2022. "Artificial intelligence in science: An emerging general method of invention," Research Policy, Elsevier, vol. 51(10).
    15. Jasjit Singh & Ajay Agrawal, 2011. "Recruiting for Ideas: How Firms Exploit the Prior Inventions of New Hires," Management Science, INFORMS, vol. 57(1), pages 129-150, January.
    16. Yusuke Oh & Koji Takahashi, 2020. "R&D and Innovation: Evidence from Patent Data," Bank of Japan Working Paper Series 20-E-7, Bank of Japan.
    17. Sitian Liu & Yichen Su, 2026. "The Geography of Research: The Trade-Off Between Knowledge Production and Access," Papers 2608.15981, arXiv.org.
    18. Stefano Bianchini & Moritz Müller & Pierre Pelletier, 2022. "Artificial intelligence in science: An emerging general method of invention," Post-Print hal-03958025, HAL.
    19. Carvalho, Vasco & Freund, Lukas B., 2026. "Firms as Foragers," CEPR Discussion Papers 21753, Centre for Economic Policy Research.

    More about this item

    JEL classification:

    • O3 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes
    • O4 - Economic Development, Innovation, Technological Change, and Growth - - Economic Growth and Aggregate Productivity
    • Z38 - Other Special Topics - - Tourism Economics - - - Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:24541. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.