IDEAS home Printed from https://ideas.repec.org/a/aea/jecper/v31y2017i4p103-24.html
   My bibliography  Save this article

Experimentation at Scale

Author

Listed:
  • Karthik Muralidharan
  • Paul Niehaus

Abstract

This paper makes the case for greater use of randomized experiments "at scale." We review various critiques of experimental program evaluation in developing countries, and discuss how experimenting at scale along three specific dimensions—the size of the sampling frame, the number of units treated, and the size of the unit of randomization—can help alleviate the concerns raised. We find that program-evaluation randomized controlled trials published over the last 15 years have typically been "small" in these senses, but also identify a number of examples—including from our own work—demonstrating that experimentation at much larger scales is both feasible and valuable.

Suggested Citation

  • Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
  • Handle: RePEc:aea:jecper:v:31:y:2017:i:4:p:103-24
    Note: DOI: 10.1257/jep.31.4.103
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/articles?id=10.1257/jep.31.4.103
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=AQDEK3GF0G74vzdUPNiQcL52AaYozkrA
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=c5OfanoRJLEQTiZVwYZmZEpOcrvo_fbt
    Download Restriction: no

    File URL: https://www.aeaweb.org/articles/attachments?retrieve=oCZcvHeW-sgm5jzHQBytprLNm_JXNUvY
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Esther Duflo & Michael Kremer & Jonathan Robinson, 2011. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 101(6), pages 2350-2390, October.
    2. Benjamin F. Jones, 2009. "The Burden of Knowledge and the "Death of the Renaissance Man": Is Innovation Getting Harder?," Review of Economic Studies, Oxford University Press, vol. 76(1), pages 283-317.
    3. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    4. Jessica Cohen & Pascaline Dupas, 2008. "Free Distribution or Cost-Sharing? Evidence from a Malaria Prevention Experiment," NBER Working Papers 14406, National Bureau of Economic Research, Inc.
    5. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    6. Benjamin Feigenberg & Erica Field & Rohini Pande, 2013. "The Economic Returns to Social Interaction: Experimental Evidence from Microfinance," Review of Economic Studies, Oxford University Press, vol. 80(4), pages 1459-1483.
    7. Dean Karlan & Jonathan Zinman, 2009. "Observing Unobservables: Identifying Information Asymmetries With a Consumer Credit Field Experiment," Econometrica, Econometric Society, vol. 77(6), pages 1993-2008, November.
    8. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    9. Henrik Jacobsen Kleven & Martin B. Knudsen & Claus Thustrup Kreiner & Søren Pedersen & Emmanuel Saez, 2011. "Unwilling or Unable to Cheat? Evidence From a Tax Audit Experiment in Denmark," Econometrica, Econometric Society, vol. 79(3), pages 651-692, May.
    10. Jesse M Cunha & Giacomo De Giorgi & Seema Jayachandran, 2019. "The Price Effects of Cash Versus In-Kind Transfers," Review of Economic Studies, Oxford University Press, vol. 86(1), pages 240-281.
    11. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    12. Nava Ashraf & James Berry & Jesse M. Shapiro, 2010. "Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia," American Economic Review, American Economic Association, vol. 100(5), pages 2383-2413, December.
    13. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
    14. Orazio P. Attanasio & Costas Meghir & Ana Santiago, 2012. "Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to Evaluate PROGRESA," Review of Economic Studies, Oxford University Press, vol. 79(1), pages 37-66.
    15. repec:oup:qjecon:v:133:y:2018:i:2:p:993-1039. is not listed on IDEAS
    16. Tessa Bold & Mwangi Kimenyi & Germano Mwabu & Alice Ng'ang'a & Justin Sandefur, 2013. "Scaling-up What Works: Experimental Evidence on External Validity in Kenyan Education," CSAE Working Paper Series 2013-04, Centre for the Study of African Economies, University of Oxford.
    17. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    18. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    19. Leonardo Bursztyn & Florian Ederer & Bruno Ferman & Noam Yuchtman, 2014. "Understanding Mechanisms Underlying Peer Effects: Evidence From a Field Experiment on Financial Decisions," Econometrica, Econometric Society, vol. 82(4), pages 1273-1301, July.
    20. Vivi Alatas & Abhijit Banerjee & Rema Hanna & Benjamin A. Olken & Julia Tobias, 2012. "Targeting the Poor: Evidence from a Field Experiment in Indonesia," American Economic Review, American Economic Association, vol. 102(4), pages 1206-1240, June.
    21. Karthik Muralidharan & Paul Niehaus & Sandip Sukhtankar, 2016. "Building State Capacity: Evidence from Biometric Smartcards in India," American Economic Review, American Economic Association, vol. 106(10), pages 2895-2929, October.
    22. Joppe de Ree & Karthik Muralidharan & Menno Pradhan & Halsey Rogers, 2018. "Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia," The Quarterly Journal of Economics, Oxford University Press, vol. 133(2), pages 993-1039.
    23. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    24. Karthik Muralidharan & Paul Niehaus & Sandip Sukhtankar, 2017. "General Equilibrium Effects of (Improving) Public Employment Programs: Experimental Evidence from India," NBER Working Papers 23838, National Bureau of Economic Research, Inc.
    25. Alessandro Tarozzi & Aprajit Mahajan & Brian Blackburn & Dan Kopf & Lakshmi Krishnan & Joanne Yoong, 2014. "Micro-loans, Insecticide-Treated Bednets, and Malaria: Evidence from a Randomized Controlled Trial in Orissa, India," American Economic Review, American Economic Association, vol. 104(7), pages 1909-1941, July.
    26. Nava Ashraf & Erica Field & Jean Lee, 2014. "Household Bargaining and Excess Fertility: An Experimental Study in Zambia," American Economic Review, American Economic Association, vol. 104(7), pages 2210-2237, July.
    27. Paul Schultz, T., 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program," Journal of Development Economics, Elsevier, vol. 74(1), pages 199-250, June.
    28. Andrew Beath & Fotini Christia & Georgy Egorov & Ruben Enikolopov, 2014. "Electoral Rules and the Quality of Politicians: Theory and Evidence from a Field Experiment in Afghanistan," NBER Working Papers 20082, National Bureau of Economic Research, Inc.
    29. Kenneth I. Wolpin & Petra E. Todd, 2006. "Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility," American Economic Review, American Economic Association, vol. 96(5), pages 1384-1417, December.
    30. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2018. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," NBER Working Papers 24876, National Bureau of Economic Research, Inc.
    31. Wolpin, Kenneth I., 2013. "The Limits of Inference without Theory," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262019086, March.
    32. Leonardo Bursztyn & Lucas C. Coffman, 2012. "The Schooling Decision: Family Preferences, Intergenerational Conflict, and Moral Hazard in the Brazilian Favelas," Journal of Political Economy, University of Chicago Press, vol. 120(3), pages 359-397.
    33. repec:aea:jecper:v:31:y:2017:i:2:p:33-58 is not listed on IDEAS
    34. Nava Ashraf & Dean Karlan & Wesley Yin, 2006. "Tying Odysseus to the Mast: Evidence From a Commitment Savings Product in the Philippines," The Quarterly Journal of Economics, Oxford University Press, vol. 121(2), pages 635-672.
    35. Raghabendra Chattopadhyay & Esther Duflo, 2004. "Women as Policy Makers: Evidence from a Randomized Policy Experiment in India," Econometrica, Econometric Society, vol. 72(5), pages 1409-1443, September.
    36. Andrew Beath & Fotini Christia & Georgy Egorov & Ruben Enikolopov, 2016. "Electoral Rules and Political Selection: Theory and Evidence from a Field Experiment in Afghanistan," Review of Economic Studies, Oxford University Press, vol. 83(3), pages 932-968.
    37. Suresh de Mel & David McKenzie & Christopher Woodruff, 2009. "Returns to Capital in Microenterprises: Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 124(1), pages 423-423.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:eee:poleco:v:54:y:2018:i:c:p:167-174 is not listed on IDEAS
    2. Omar Al-Ubaydli & John List & Dana Suskind, 2019. "The science of using science: Towards an understanding of the threats to scaling experiments," Artefactual Field Experiments 00670, The Field Experiments Website.
    3. Chaisemartin, Clement de & Navarrete, Nicolas, 2019. "The direct and spillover effects of a mental health program for disruptive students," CAGE Online Working Paper Series 401, Competitive Advantage in the Global Economy (CAGE).
    4. Andrew Dustan & Stanislao Maldonado & Juan Manuel Hernandez-Agramonte, 2018. "Motivating bureaucrats with non-monetary incentives when state capacity is weak: Evidence from large-scale field experiments in Peru," Working Papers 136, Peruvian Economic Association.
    5. Andrew Dustan & Juan Manuel Hernandez-Agramonte & Stanislao Maldonado, 2018. "Motivating bureaucrats with non-monetary incentives when state capacity is weak: Evidence from large-scale," Natural Field Experiments 00664, The Field Experiments Website.
    6. Eszter Czibor & David Jimenez-Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," NBER Working Papers 25451, National Bureau of Economic Research, Inc.
    7. Dalton, Patricio & Rüschenpöhler, Julius & Zia, Bilal, 2018. "Determinants and Dynamics of Business Aspirations : Evidence from Small-scale Entrepreneurs in an Emerging Market," Discussion Paper 2018-009, Tilburg University, Center for Economic Research.
    8. Cameron, Lisa & Olivia, Susan & Shah, Manisha, 2019. "Scaling up sanitation: Evidence from an RCT in Indonesia," Journal of Development Economics, Elsevier, vol. 138(C), pages 1-16.

    More about this item

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • D78 - Microeconomics - - Analysis of Collective Decision-Making - - - Positive Analysis of Policy Formulation and Implementation
    • O10 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:jecper:v:31:y:2017:i:4:p:103-24. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Michael P. Albert). General contact details of provider: http://edirc.repec.org/data/aeaaaea.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.