IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Context Matters for Size: Why External Validity Claims and Development Practice Don't Mix-Working Paper 336

  • Lant Pritchett, Justin Sandefur

    ()

In this paper we examine how policymakers and practitioners should interpret the impact evaluation literature when presented with conflicting experimental and non-experimental estimates of the same intervention across varying contexts. We show three things. First, as is well known, non-experimental estimates of a treatment effect comprise a causal treatment effect and a bias term due to endogenous selection into treatment. When non-experimental estimates vary across contexts any claim for external validity of an experimental result must make the assumption that (a) treatment effects are constant across contexts, while (b) selection processes vary across contexts. This assumption is rarely stated or defended in systematic reviews of evidence. Second, as an illustration of these issues, we examine two thoroughly researched literatures in the economics of education—class size effects and gains from private schooling—which provide experimental and non-experimental estimates of causal effects from the same context and across multiple contexts. We show that the range of “true” causal effects in these literatures implies OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Third, we show that in important cases in economics, parameter heterogeneity is driven by economy- or institution-wide contextual factors, rather than personal characteristics, making it difficult to overcome external validity concerns through estimation of heterogeneous treatment effects within a single localized sample. We conclude with recommendations for research and policy, including the need to evaluate programs in context, and avoid simple analogies to clinical medicine in which “systematic reviews” attempt to identify best-practices by putting most (or all) weight on the most “rigorous” evidence with no allowance for context.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://www.cgdev.org/sites/default/files/context-matters-for-size_0.pdf
Download Restriction: no

Paper provided by Center for Global Development in its series Working Papers with number 336.

as
in new window

Length: 55 pages
Date of creation: Aug 2013
Date of revision:
Handle: RePEc:cgd:wpaper:336
Contact details of provider: Web page: http://www.cgdev.org

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Miguel Urquiola & Eric A. Verhoogen, 2007. "Class size and sorting in market equilibrium: Theory and evidence," Discussion Papers 0607-14, Columbia University, Department of Economics.
  2. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
  3. Brodeur, Abel & Lé, Mathias & Sangnier, Marc & Zylberberg, Yanos, 2013. "Star Wars: The Empirics Strike Back," IZA Discussion Papers 7268, Institute for the Study of Labor (IZA).
  4. Shahrukh Rafi Khan & David Kiefer, 2007. "Educational Production Functions for Rural Pakistan: A Comparative Institutional Analysis," Education Economics, Taylor & Francis Journals, vol. 15(3), pages 327-342.
  5. Banerjee, Abhijit & Cole, Shawn & Duflo, Esther & Linden, Leigh, 2006. "Remedying Education: Evidence from Two Randomized Experiments in India," CEPR Discussion Papers 5446, C.E.P.R. Discussion Papers.
  6. Behrman, Jere R. & Khan, Shahrukh & Ross, David & Sabot, Richard, 1997. "School quality and cognitive achievement production: A case study for rural Pakistan," Economics of Education Review, Elsevier, vol. 16(2), pages 127-142, April.
  7. M. Niaz Asadullah, 2005. "The effect of class size on student achievement: evidence from Bangladesh," Applied Economics Letters, Taylor & Francis Journals, vol. 12(4), pages 217-221.
  8. Alan B. Krueger, 1999. "Experimental Estimates Of Education Production Functions," The Quarterly Journal of Economics, MIT Press, vol. 114(2), pages 497-532, May.
  9. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
  10. Michaelowa, Katharina, 2001. "Primary Education Quality in Francophone Sub-Saharan Africa: Determinants of Learning Achievement and Efficiency Considerations," World Development, Elsevier, vol. 29(10), pages 1699-1716, October.
  11. Tessa Bold & Mwangi Kimenyi & Germano Mwabu & Alice Ng'ang'a & Justin Sandefur, 2013. "Scaling-up What Works: Experimental Evidence on External Validity in Kenyan Education," Economics Series Working Papers WPS/2013-04, University of Oxford, Department of Economics.
  12. Brown, Philip H. & Park, Albert, 2002. "Education and poverty in rural China," Economics of Education Review, Elsevier, vol. 21(6), pages 523-541, December.
  13. Tessa Bold, Mwangi Kimenyi, Germano Mwabu, Justin Sandefur, 2011. " The High Return to Private Schooling in a Low-Income Country- Working Paper 279," Working Papers 279, Center for Global Development.
  14. Rodrik, Dani, 2008. "The New Development Economics: We Shall Experiment, but How Shall We Learn?," Working Paper Series rwp08-055, Harvard University, John F. Kennedy School of Government.
  15. Bacolod, Marigee P. & Tobias, Justin L., 2006. "Schools, school quality and achievement growth: Evidence from the Philippines," Economics of Education Review, Elsevier, vol. 25(6), pages 619-632, December.
  16. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
  17. Miguel Urquiola, 2006. "Identifying Class Size Effects in Developing Countries: Evidence from Rural Bolivia," The Review of Economics and Statistics, MIT Press, vol. 88(1), pages 171-177, February.
  18. Pritchett, Lant & Samji, Salimah & Hammer, Jeffrey, 2012. "It.s All About MeE: Using Structured Experiential Learning (.e.) to Crawl the Design Space," Working Paper Series UNU-WIDER Research Paper , World Institute for Development Economic Research (UNU-WIDER).
  19. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, 04.
  20. Bedi, Arjun S & Marshall, Jeffrey H, 1999. "School Attendance and Student Achievement: Evidence from Rural Honduras," Economic Development and Cultural Change, University of Chicago Press, vol. 47(3), pages 657-82, April.
  21. Cristian Pop-Eleches, 2010. "The Supply of Birth Control Methods, Education, and Fertility: Evidence from Romania," Journal of Human Resources, University of Wisconsin Press, vol. 45(4), pages 971-997.
  22. Angus Deaton, 2010. "Instruments, randomization, and learning about development," Working Papers 1224, Princeton University, Woodrow Wilson School of Public and International Affairs, Research Program in Development Studies..
  23. Daniel Suryadarma & Asep Suryahadi & Sudarno Sumarto & F. Halsey Rogers, 2006. "Improving Student Performance in Public Primary Schools in Developing Countries: Evidence from Indonesia," Education Economics, Taylor & Francis Journals, vol. 14(4), pages 401-429.
  24. Guildo W. Imbens, 2003. "Sensitivity to Exogeneity Assumptions in Program Evaluation," American Economic Review, American Economic Association, vol. 93(2), pages 126-132, May.
  25. Monazza Aslam, 2003. "The Determinants of Student Achievement in Government and Private Schools in Pakistan," The Pakistan Development Review, Pakistan Institute of Development Economics, vol. 42(4), pages 841-876.
  26. Lant Pritchett & Michael Woolcock & Matt Andrews, 2013. "Looking Like a State: Techniques of Persistent Failure in State Capability for Implementation," Journal of Development Studies, Taylor & Francis Journals, vol. 49(1), pages 1-18, January.
  27. Esther Duflo, 2001. "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment," American Economic Review, American Economic Association, vol. 91(4), pages 795-813, September.
  28. Chin, Aimee, 2005. "Can redistributing teachers across schools raise educational attainment? Evidence from Operation Blackboard in India," Journal of Development Economics, Elsevier, vol. 78(2), pages 384-405, December.
  29. Marshall, Jeffery H., 2009. "School quality and learning gains in rural Guatemala," Economics of Education Review, Elsevier, vol. 28(2), pages 207-216, April.
  30. Alexander Tabarrok, 2013. "Private Education In India: A Novel Test Of Cream Skimming," Contemporary Economic Policy, Western Economic Association International, vol. 31(1), pages 1-12, 01.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:cgd:wpaper:336. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (David Roodman)

The email address of this maintainer does not seem to be valid anymore. Please ask David Roodman to update the entry or send us the correct address

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.