IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/23698.html
   My bibliography  Save this paper

Inside Job or Deep Impact? Using Extramural Citations to Assess Economic Scholarship

Author

Listed:
  • Joshua Angrist
  • Pierre Azoulay
  • Glenn Ellison
  • Ryan Hill
  • Susan Feng Lu

Abstract

Does academic economic research produce material of scientific value, or are academic economists writing only for clients and peers? Is economics scholarship uniquely insular? We address these questions by quantifying interactions between economics and other disciplines. Changes in the impact of economic scholarship are measured here by the way other disciplines cite us. We document a clear rise in the extramural influence of economic research, while also showing that economics is increasingly likely to reference other social sciences. A breakdown of extramural citations by economics fields shows broad field impact. Differentiating between theoretical and empirical papers classified using machine learning, we see that much of the rise in economics’ extramural influence reflects growth in citations to empirical work. This parallels a growing share of empirical cites within economics. At the same time, the disciplines of computer science and operations research are mostly influenced by economic theory.

Suggested Citation

  • Joshua Angrist & Pierre Azoulay & Glenn Ellison & Ryan Hill & Susan Feng Lu, 2017. "Inside Job or Deep Impact? Using Extramural Citations to Assess Economic Scholarship," NBER Working Papers 23698, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:23698
    Note: IO LS PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w23698.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Matthew T. Panhans & John D. Singleton, 2015. "The Empirical Economist's Toolkit: From Models to Methods," Center for the History of Political Economy Working Paper Series 2015-3, Center for the History of Political Economy.
    2. Hall, Robert E, 1978. "Stochastic Implications of the Life Cycle-Permanent Income Hypothesis: Theory and Evidence," Journal of Political Economy, University of Chicago Press, vol. 86(6), pages 971-987, December.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Beatrice Cherrier, 2017. "Classifying Economics: A History of the JEL Codes," Journal of Economic Literature, American Economic Association, vol. 55(2), pages 545-579, June.
    5. Lawrence J. Christiano & Martin Eichenbaum & Charles L. Evans, 2005. "Nominal Rigidities and the Dynamic Effects of a Shock to Monetary Policy," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 1-45, February.
    6. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    7. Daniel S. Hamermesh, 2013. "Six Decades of Top Economics Publishing: Who and How?," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 162-172, March.
    8. Michael A. Kelly & Stephen Bruestle, 2011. "Trend Of Subjects Published In Economics Journals 1969–2007," Economic Inquiry, Western Economic Association International, vol. 49(3), pages 658-673, July.
    9. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    10. Robert E. Hall & Charles I. Jones, 1999. "Why do Some Countries Produce So Much More Output Per Worker than Others?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(1), pages 83-116.
    11. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    12. Roger Backhouse & Beatrice Cherrier, 2014. "Becoming Applied: The Transformation of Economics after 1970," Discussion Papers 14-11, Department of Economics, University of Birmingham.
    13. Rik Pieters & Hans Baumgartner, 2002. "Who Talks to Whom? Intra- and Interdisciplinary Communication of Economics Journals," Journal of Economic Literature, American Economic Association, vol. 40(2), pages 483-509, June.
    14. Berry, Steven & Levinsohn, James & Pakes, Ariel, 1995. "Automobile Prices in Market Equilibrium," Econometrica, Econometric Society, vol. 63(4), pages 841-890, July.
    15. Loet Leydesdorff, 2004. "Top-down decomposition of the Journal Citation Reportof the Social Science Citation Index: Graph- and factor-analytical approaches," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(2), pages 159-180, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    2. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    3. Miguel Atienza & Guillermo Armando Ronda-Pupo & Nicholas Phelps, 2019. "Bridges over troubled water? Journals, geographers and economists in the field of economy and space 1980–2017," Environment and Planning A, , vol. 51(8), pages 1800-1823, November.
    4. Kang, Yankun & Leng, Xuan & Liao, Yunxiang & Zheng, Shilin, 2024. "Information disclosure, spillovers, and knowledge accumulation," China Economic Review, Elsevier, vol. 84(C).
    5. John Gibson, 2021. "The micro‐geography of academic research: How distinctive is economics?," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(4), pages 467-484, September.
    6. Kim‐Sau Chung & Meng‐Yu Liang & Melody Lo, 2022. "On the information contents of indirect citations," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 55(1), pages 156-173, February.
    7. Battistin, Erich & Ovidi, Marco, 2017. "Rising Stars," IZA Discussion Papers 11198, Institute of Labor Economics (IZA).
    8. Koffi, Marlene, 2021. "Innovative ideas and gender inequality," CLEF Working Paper Series 35, Canadian Labour Economics Forum (CLEF), University of Waterloo.
    9. Antonin Macé, 2023. "The Limits of Citation Counts," Working Papers halshs-01630095, HAL.
    10. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    11. Olivér Kovács, 2019. "Grounding Complexity Economics in Framing Modern Governance," Acta Oeconomica, Akadémiai Kiadó, Hungary, vol. 69(4), pages 571-594, December.
    12. Peter Andre & Armin Falk, 2021. "Policy relevant, multidisciplinary, disruptive: What kind of research do economists want?," ECONtribute Policy Brief Series 023, University of Bonn and University of Cologne, Germany.
    13. Kyle Siler & Philippe Vincent-Lamarre & Cassidy R Sugimoto & Vincent Larivière, 2022. "Cumulative advantage and citation performance of repeat authors in scholarly journals," PLOS ONE, Public Library of Science, vol. 17(4), pages 1-17, April.
    14. Keith H. Coble, 2020. "Relevant and/or Elegant Economics," American Journal of Agricultural Economics, John Wiley & Sons, vol. 102(2), pages 392-399, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Josh Angrist & Pierre Azoulay & Glenn Ellison & Ryan Hill & Susan Feng Lu, 2020. "Inside Job or Deep Impact? Extramural Citations and the Influence of Economic Scholarship," Journal of Economic Literature, American Economic Association, vol. 58(1), pages 3-52, March.
    2. Michel De Vroey & Luca Pensieroso, 2021. "Grounded in Methodology, Certified by Journals: The Rise and Evolution of a Mainstream in Economics," LIDAM Discussion Papers IRES 2021015, Université catholique de Louvain, Institut de Recherches Economiques et Sociales (IRES).
    3. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    4. Jeff E. Biddle & Daniel S. Hamermesh, 2016. "Theory and Measurement: Emergence, Consolidation and Erosion of a Consensus," NBER Working Papers 22253, National Bureau of Economic Research, Inc.
    5. Seltzer, Andrew J. & Hamermesh, Daniel S., 2018. "Co-authorship in economic history and economics: Are we any different?," Explorations in Economic History, Elsevier, vol. 69(C), pages 102-109.
    6. Daron Acemoglu, 2005. "Constitutions, Politics and Economics: A Review Essay on Persson and Tabellini's "The Economic Effect of Constitutions"," NBER Working Papers 11235, National Bureau of Economic Research, Inc.
    7. Joshua D. Angrist & Jörn-Steffen Pischke, 2017. "Undergraduate Econometrics Instruction: Through Our Classes, Darkly," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 125-144, Spring.
    8. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    9. José Edwards & Yann Giraud & Christophe Schinckus, 2018. "A quantitative turn in the historiography of economics?," Journal of Economic Methodology, Taylor & Francis Journals, vol. 25(4), pages 283-290, October.
    10. Lutz Bornmann & Klaus Wohlrabe, 2024. "Recent Temporal Dynamics in Economics: Empirical Analyses of Annual Publications in Economic Fields," CESifo Working Paper Series 10881, CESifo.
    11. Püttmann, Vitus & Thomsen, Stephan L. & Trunzer, Johannes, 2020. "Zur Relevanz von Ausstattungsunterschieden für Forschungsleistungsvergleiche: Ein Diskussionsbeitrag für die Wirtschaftswissenschaften in Deutschland," Hannover Economic Papers (HEP) dp-679, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät, revised Mar 2021.
    12. Andrei Dubovik & Clemens Fiedler & Alexei Parakhonyak, 2022. "Temporal Patterns in Economics Research," CPB Discussion Paper 440, CPB Netherlands Bureau for Economic Policy Analysis.
    13. Michel De Vroey & Luca Pensieroso, 2016. "The Rise of a Mainstream in Economics," LIDAM Discussion Papers IRES 2016026, Université catholique de Louvain, Institut de Recherches Economiques et Sociales (IRES).
    14. Isaiah Andrews & Matthew Gentzkow & Jesse M. Shapiro, 2017. "Measuring the Sensitivity of Parameter Estimates to Estimation Moments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(4), pages 1553-1592.
    15. Martina Cioni & Giovanni Federico & Michelangelo Vasta, 2023. "Is economic history changing its nature? Evidence from top journals," Cliometrica, Journal of Historical Economics and Econometric History, Association Française de Cliométrie (AFC), vol. 17(1), pages 23-48, January.
    16. Marion Fourcade & Etienne Ollion & Yann Algan, 2015. "La superioridad de los economistas," Revista de Economía Institucional, Universidad Externado de Colombia - Facultad de Economía, vol. 17(33), pages 13-43, July-Dece.
    17. Lea Kosnik, 2016. "JEL Codes: What Do They Mean and Are They Used Consistently?," Working Papers 1011, University of Missouri-St. Louis, Department of Economics.
    18. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    19. Chul‐Woo Kwon & Peter F. Orazem & Daniel M. Otto, 2006. "Off‐farm labor supply responses to permanent and transitory farm income," Agricultural Economics, International Association of Agricultural Economists, vol. 34(1), pages 59-67, January.
    20. Xiaohong Chen & Andres Santos, 2018. "Overidentification in Regular Models," Econometrica, Econometric Society, vol. 86(5), pages 1771-1817, September.

    More about this item

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • A12 - General Economics and Teaching - - General Economics - - - Relation of Economics to Other Disciplines
    • A13 - General Economics and Teaching - - General Economics - - - Relation of Economics to Social Values
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:23698. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.