External Validity and Partner Selection Bias
AbstractProgram evaluation often involves generalizing internally-valid site-specific estimates to a different target population. While many analyses have tested the assumptions required for internal validity (e.g. LaLonde 1986), there has been little empirical assessment of external validity in any context, because identical treatments are rarely evaluated in multiple sites. This paper examines a remarkable series of 14 energy conservation field experiments run by a company called Opower, involving 550,000 households in different cities across the U.S. We first show that the site-specific treatment effect heterogeneity is both statistically and economically significant. We then show that Opower partners are selected on partner-level characteristics that are correlated with the treatment effect. This "partner selection bias" implies that replications with additional partners have not given an unbiased estimate of the distribution of treatment effects in non-partner sites. We augment these results in a different context by showing that partner microfinance institutions (MFIs) that carry out randomized experiments are selected on observable characteristics from the global pool of MFIs. Finally, we propose two simple suggestive tests of external validity that can be used in the absence of data from many sites: comparison of observable sample and target site characteristics and an F-test of heterogeneous treatment effects across "sub-sites" within a site.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by National Bureau of Economic Research, Inc in its series NBER Working Papers with number 18373.
Date of creation: Sep 2012
Date of revision:
Note: EEE LS
Contact details of provider:
Postal: National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.
Web page: http://www.nber.org
More information through EDIRC
Find related papers by JEL classification:
- C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
- D12 - Microeconomics - - Household Behavior - - - Consumer Economics: Empirical Analysis
- L94 - Industrial Organization - - Industry Studies: Transportation and Utilities - - - Electric Utilities
- O12 - Economic Development, Technological Change, and Growth - - Economic Development - - - Microeconomic Analyses of Economic Development
- Q41 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy - - - Demand and Supply
This paper has been announced in the following NEP Reports:
- NEP-ALL-2012-09-22 (All new papers)
- NEP-EXP-2012-09-22 (Experimental Economics)
- NEP-MFD-2012-09-22 (Microfinance)
You can help add them by filling out this form.
CitEc Project, subscribe to its RSS feed for this item.
- Allcott, Hunt & Rogers, Todd, 2012. "How Long Do Treatment Effects Last? Persistence and Durability of a Descriptive Norms Intervention's Effect on Energy Conservation," Working Paper Series rwp12-045, Harvard University, John F. Kennedy School of Government.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ().
If references are entirely missing, you can add them using this form.