Was there a Riverside miracle? An hierarchical framework for evaluating programs with grouped data
AbstractThis paper uses data from the Greater Avenues for Independence (GAIN) demonstration to discuss the evaluation of programs that are implemented at multiple sites. Two frequently used methods are pooling the data or using fixed effects (an extreme version of which estimates separate models for each site). The former approach, however, ignores site effects. Though the latter incorporates site effects, it lacks a framework for predicting the impact of subsequent implementations of the program (e.g., will a new implementation resemble Riverside or Alameda?). I present an hierarchical model that lies between these two extremes. For the GAIN data, I demonstrate that the model captures much of the site-to-site variation of treatment effects, but has less uncertainty than a model which estimates treatment effects separately for each site. I also show that uncertainty in predicting site effects is important: when the predictive uncertainty is ignored, the treatment impact for the Riverside sites is significant, but when we consider predictive uncertainty, the impact for the Riverside sites is insignificant. Finally, I demonstrate that the model is able to extrapolate site effects with reasonable accuracy, when the site for which the prediction is being made does not differ substantially from the sites already observed. For example, the San Diego treatment effects could have been predicted based on observable site characteristics, but the Riverside effects are consistently underestimated.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Columbia University, Department of Economics in its series Discussion Papers with number 0102-15.
Length: 40 pages
Date of creation: 2002
Date of revision:
Contact details of provider:
Postal: 1022 International Affairs Building, 420 West 118th Street, New York, NY 10027
Phone: (212) 854-3680
Fax: (212) 854-8059
Web page: http://www.econ.columbia.edu/
More information through EDIRC
Other versions of this item:
- Dehejia, Rajeev H, 2003. "Was There a Riverside Miracle? A Hierarchical Framework for Evaluating Programs with Grouped Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 1-11, January.
You can help add them by filling out this form.
CitEc Project, subscribe to its RSS feed for this item.
- Guido Imbens & Jeffrey Wooldridge, 2008.
"Recent developments in the econometrics of program evaluation,"
CeMMAP working papers
CWP24/08, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
- Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
- Guido M. Imbens & Jeffrey M. Wooldridge, 2008. "Recent Developments in the Econometrics of Program Evaluation," NBER Working Papers 14251, National Bureau of Economic Research, Inc.
- Imbens, Guido W. & Wooldridge, Jeffrey M., 2008. "Recent Developments in the Econometrics of Program Evaluation," IZA Discussion Papers 3640, Institute for the Study of Labor (IZA).
- Wooldridge, Jeffrey M. & Imbens, Guido, 2009. "Recent Developments in the Econometrics of Program Evaluation," Scholarly Articles 3043416, Harvard University Department of Economics.
- Arpino, Bruno & Mealli, Fabrizia, 2011.
"The specification of the propensity score in multilevel observational studies,"
Computational Statistics & Data Analysis,
Elsevier, vol. 55(4), pages 1770-1780, April.
- Bruno Arpino & Fabrizia Mealli, 2008. "The specification of the propensity score in multilevel observational studies," Working Papers 006, "Carlo F. Dondena" Centre for Research on Social Dynamics (DONDENA), Università Commerciale Luigi Bocconi.
- Arpino, Bruno & Mealli, Fabrizia, 2008. "The specification of the propensity score in multilevel observational studies," MPRA Paper 17407, University Library of Munich, Germany.
- Donald M. Pianto & Sergei Soares, 2004. "Use Of Survey Design For The Evaluation Of Social Programs: The Pnad And Peti," Anais do XXXII Encontro Nacional de Economia [Proceedings of the 32th Brazilian Economics Meeting] 133, ANPEC - Associação Nacional dos Centros de Pósgraduação em Economia [Brazilian Association of Graduate Programs in Economics].
- Carlos A. Flores & Oscar A. Mitnik, 2013.
"Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies,"
The Review of Economics and Statistics,
MIT Press, vol. 95(5), pages 1691-1707, December.
- Carlos A. Flores & Oscar A. Mitnik, 2011. "Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies," Working Papers 2011-10, University of Miami, Department of Economics.
- V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006.
"Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Re-Analysis of the California GAIN Program,"
NBER Working Papers
11939, National Bureau of Economic Research, Inc.
- V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Reanalysis of the California GAIN Program," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 521-566, July.
- Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
- Dehejia, Rajeev, 2013. "The porous dialectic: Experimental and non-experimental methods in development economics," Working Paper Series UNU-WIDER Research Paper , World Institute for Development Economic Research (UNU-WIDER).
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Discussion Paper Coordinator).
If references are entirely missing, you can add them using this form.