Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons
This paper analyzes 12 recent within-study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression-discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff. We identify three other studies where the quasi-experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within-study comparisons, most involving job training. Reasons for this difference are discussed. © 2008 by the Association for Public Policy Analysis and Management.
Volume (Year): 27 (2008)
Issue (Month): 4 ()
|Contact details of provider:|| Web page: http://www3.interscience.wiley.com/journal/34787/home|
References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Steven Glazerman Dan Levy David Myers, 2003. "Nonexperimental Versus Experimental Estimates of Earnings Impacts," Mathematica Policy Research Reports 7c8bd68ac8db47caa57c70ee1, Mathematica Policy Research.
- McKenzie, David & Gibson, John & Stillman, Steven, 2006.
"How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration,"
IZA Discussion Papers
2087, Institute for the Study of Labor (IZA).
- David McKenzie & Steven Stillman & John Gibson, 2010. "How Important is Selection? Experimental VS. Non‐Experimental Measures of the Income Gains from Migration," Journal of the European Economic Association, European Economic Association, vol. 8(4), pages 913-945, 06.
- David McKenzie & John Gibson & Steven Stillman, 2010. "How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration," Journal of the European Economic Association, MIT Press, vol. 8(4), pages 913-945, 06.
- James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
- James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
- repec:mpr:mprres:3694 is not listed on IDEAS
- Jeffrey Smith & Petra Todd, 2003.
"Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?,"
University of Western Ontario, Centre for Human Capital and Productivity (CHCP) Working Papers
20035, University of Western Ontario, Centre for Human Capital and Productivity (CHCP).
- A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
- Cook, Thomas D., 2008. ""Waiting for Life to Arrive": A history of the regression-discontinuity design in Psychology, Statistics and Economics," Journal of Econometrics, Elsevier, vol. 142(2), pages 636-654, February.
- LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
- Roberto Agodini & Mark Dynarski, 2004. "Are Experiments the Only Option? A Look at Dropout Prevention Programs," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 180-194, February.
When requesting a correction, please mention this item's handle: RePEc:wly:jpamgt:v:27:y:2008:i:4:p:724-750. See general information about how to correct material in RePEc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley-Blackwell Digital Licensing)or (Christopher F. Baum)
If references are entirely missing, you can add them using this form.