IDEAS home Printed from https://ideas.repec.org/a/aea/jecper/v31y2017i4p103-24.html
   My bibliography  Save this article

Experimentation at Scale

Author

Listed:
  • Karthik Muralidharan
  • Paul Niehaus

Abstract

This paper makes the case for greater use of randomized experiments "at scale." We review various critiques of experimental program evaluation in developing countries, and discuss how experimenting at scale along three specific dimensions—the size of the sampling frame, the number of units treated, and the size of the unit of randomization—can help alleviate the concerns raised. We find that program-evaluation randomized controlled trials published over the last 15 years have typically been "small" in these senses, but also identify a number of examples—including from our own work—demonstrating that experimentation at much larger scales is both feasible and valuable.

Suggested Citation

  • Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
  • Handle: RePEc:aea:jecper:v:31:y:2017:i:4:p:103-24
    Note: DOI: 10.1257/jep.31.4.103
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/articles?id=10.1257/jep.31.4.103
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=AQDEK3GF0G74vzdUPNiQcL52AaYozkrA
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=c5OfanoRJLEQTiZVwYZmZEpOcrvo_fbt
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=oCZcvHeW-sgm5jzHQBytprLNm_JXNUvY
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Esther Duflo & Michael Kremer & Jonathan Robinson, 2011. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 101(6), pages 2350-2390, October.
    2. Benjamin F. Jones, 2009. "The Burden of Knowledge and the "Death of the Renaissance Man": Is Innovation Getting Harder?," Review of Economic Studies, Oxford University Press, vol. 76(1), pages 283-317.
    3. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    4. Jessica Cohen & Pascaline Dupas, 2008. "Free Distribution or Cost-Sharing? Evidence from a Malaria Prevention Experiment," NBER Working Papers 14406, National Bureau of Economic Research, Inc.
    5. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    6. Benjamin Feigenberg & Erica Field & Rohini Pande, 2013. "The Economic Returns to Social Interaction: Experimental Evidence from Microfinance," Review of Economic Studies, Oxford University Press, vol. 80(4), pages 1459-1483.
    7. Dean Karlan & Jonathan Zinman, 2009. "Observing Unobservables: Identifying Information Asymmetries With a Consumer Credit Field Experiment," Econometrica, Econometric Society, vol. 77(6), pages 1993-2008, November.
    8. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    9. Henrik Jacobsen Kleven & Martin B. Knudsen & Claus Thustrup Kreiner & Søren Pedersen & Emmanuel Saez, 2011. "Unwilling or Unable to Cheat? Evidence From a Tax Audit Experiment in Denmark," Econometrica, Econometric Society, vol. 79(3), pages 651-692, May.
    10. Jesse M Cunha & Giacomo De Giorgi & Seema Jayachandran, 2019. "The Price Effects of Cash Versus In-Kind Transfers," Review of Economic Studies, Oxford University Press, vol. 86(1), pages 240-281.
    11. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    12. Nava Ashraf & James Berry & Jesse M. Shapiro, 2010. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia," American Economic Review, American Economic Association, vol. 100(5), pages 2383-2413, December.
    13. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
    14. Orazio P. Attanasio & Costas Meghir & Ana Santiago, 2012. "Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to Evaluate PROGRESA," Review of Economic Studies, Oxford University Press, vol. 79(1), pages 37-66.
    15. Eva Vivalt, 2015. "Heterogeneous Treatment Effects in Impact Evaluation," American Economic Review, American Economic Association, vol. 105(5), pages 467-470, May.
    16. Joppe de Ree & Karthik Muralidharan & Menno Pradhan & Halsey Rogers, 2018. "Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia," The Quarterly Journal of Economics, Oxford University Press, vol. 133(2), pages 993-1039.
    17. Tessa Bold & Mwangi Kimenyi & Germano Mwabu & Alice Ng'ang'a & Justin Sandefur, 2013. "Scaling-up What Works: Experimental Evidence on External Validity in Kenyan Education," CSAE Working Paper Series 2013-04, Centre for the Study of African Economies, University of Oxford.
    18. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    19. Suresh de Mel & David McKenzie & Christopher Woodruff, 2009. "Returns to Capital in Microenterprises: Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 124(1), pages 423-423.
    20. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    21. Leonardo Bursztyn & Florian Ederer & Bruno Ferman & Noam Yuchtman, 2014. "Understanding Mechanisms Underlying Peer Effects: Evidence From a Field Experiment on Financial Decisions," Econometrica, Econometric Society, vol. 82(4), pages 1273-1301, July.
    22. Vivi Alatas & Abhijit Banerjee & Rema Hanna & Benjamin A. Olken & Julia Tobias, 2012. "Targeting the Poor: Evidence from a Field Experiment in Indonesia," American Economic Review, American Economic Association, vol. 102(4), pages 1206-1240, June.
    23. Karthik Muralidharan & Paul Niehaus & Sandip Sukhtankar, 2016. "Building State Capacity: Evidence from Biometric Smartcards in India," American Economic Review, American Economic Association, vol. 106(10), pages 2895-2929, October.
    24. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    25. Karthik Muralidharan & Paul Niehaus & Sandip Sukhtankar, 2017. "General Equilibrium Effects of (Improving) Public Employment Programs: Experimental Evidence from India," NBER Working Papers 23838, National Bureau of Economic Research, Inc.
    26. Nava Ashraf & Dean Karlan & Wesley Yin, 2006. "Tying Odysseus to the Mast: Evidence From a Commitment Savings Product in the Philippines," The Quarterly Journal of Economics, Oxford University Press, vol. 121(2), pages 635-672.
    27. Alessandro Tarozzi & Aprajit Mahajan & Brian Blackburn & Dan Kopf & Lakshmi Krishnan & Joanne Yoong, 2014. "Micro-loans, Insecticide-Treated Bednets, and Malaria: Evidence from a Randomized Controlled Trial in Orissa, India," American Economic Review, American Economic Association, vol. 104(7), pages 1909-1941, July.
    28. Nava Ashraf & Erica Field & Jean Lee, 2014. "Household Bargaining and Excess Fertility: An Experimental Study in Zambia," American Economic Review, American Economic Association, vol. 104(7), pages 2210-2237, July.
    29. Rema Hanna & Sendhil Mullainathan & Joshua Schwartzstein, 2014. "Learning Through Noticing: Theory and Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 129(3), pages 1311-1353.
    30. Jere R. Behrman & Susan W. Parker & Petra E. Todd & Kenneth I. Wolpin, 2015. "Aligning Learning Incentives of Students and Teachers: Results from a Social Experiment in Mexican High Schools," Journal of Political Economy, University of Chicago Press, vol. 123(2), pages 325-364.
    31. Paul Schultz, T., 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program," Journal of Development Economics, Elsevier, vol. 74(1), pages 199-250, June.
    32. Andrew Beath & Fotini Christia & Georgy Egorov & Ruben Enikolopov, 2014. "Electoral Rules and the Quality of Politicians: Theory and Evidence from a Field Experiment in Afghanistan," NBER Working Papers 20082, National Bureau of Economic Research, Inc.
    33. Kenneth I. Wolpin & Petra E. Todd, 2006. "Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility," American Economic Review, American Economic Association, vol. 96(5), pages 1384-1417, December.
    34. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, Oxford University Press, vol. 134(3), pages 1627-1673.
    35. Wolpin, Kenneth I., 2013. "The Limits of Inference without Theory," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262019086.
    36. Leonardo Bursztyn & Lucas C. Coffman, 2012. "The Schooling Decision: Family Preferences, Intergenerational Conflict, and Moral Hazard in the Brazilian Favelas," Journal of Political Economy, University of Chicago Press, vol. 120(3), pages 359-397.
    37. Hamish Low & Costas Meghir, 2017. "The Use of Structural Models in Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 33-58, Spring.
    38. Raghabendra Chattopadhyay & Esther Duflo, 2004. "Women as Policy Makers: Evidence from a Randomized Policy Experiment in India," Econometrica, Econometric Society, vol. 72(5), pages 1409-1443, September.
    39. Andrew Beath & Fotini Christia & Georgy Egorov & Ruben Enikolopov, 2016. "Electoral Rules and Political Selection: Theory and Evidence from a Field Experiment in Afghanistan," Review of Economic Studies, Oxford University Press, vol. 83(3), pages 932-968.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.
    2. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," World Bank Research Observer, World Bank Group, vol. 33(1), pages 34-64.
    3. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.
    4. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    5. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    6. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    7. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    8. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    9. Sebastian Galiani & Juan Pantano, 2021. "Structural Models: Inception and Frontier," NBER Working Papers 28698, National Bureau of Economic Research, Inc.
    10. Abhijit V. Banerjee & Esther Duflo, 2010. "Giving Credit Where It Is Due," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 61-80, Summer.
    11. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    12. Francesca Marchetta & Tom Dilly, 2019. "Supporting Education in Africa: Opportunities and Challenges for an Impact Investor," Working Papers hal-02288103, HAL.
    13. Teresa Molina Millan & Karen Macours, 2017. "Attrition in randomized control trials: Using tracking information to correct bias," NOVAFRICA Working Paper Series wp1702, Universidade Nova de Lisboa, Nova School of Business and Economics, NOVAFRICA.
    14. Meredith, Jennifer & Robinson, Jonathan & Walker, Sarah & Wydick, Bruce, 2013. "Keeping the doctor away: Experimental evidence on investment in preventative health products," Journal of Development Economics, Elsevier, vol. 105(C), pages 196-210.
    15. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    16. Richard Akresh & Damien de Walque & Harounan Kazianga, 2013. "Cash Transfers and Child Schooling: Evidence from a Randomized Evaluation of the Role of Conditionality," Economics Working Paper Series 1301, Oklahoma State University, Department of Economics and Legal Studies in Business.
    17. Pieter Gautier & Paul Muller & Bas van der Klaauw & Michael Rosholm & Michael Svarer, 2018. "Estimating Equilibrium Effects of Job Search Assistance," Journal of Labor Economics, University of Chicago Press, vol. 36(4), pages 1073-1125.
    18. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    19. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    20. William Easterly, 2009. "Can the West Save Africa?," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 373-447, June.

    More about this item

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • D78 - Microeconomics - - Analysis of Collective Decision-Making - - - Positive Analysis of Policy Formulation and Implementation
    • O10 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:jecper:v:31:y:2017:i:4:p:103-24. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.