Can we trust shoestring evaluations ?
AbstractMany more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal,"shoestring,"method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by The World Bank in its series Policy Research Working Paper Series with number 5983.
Date of creation: 01 Mar 2012
Date of revision:
Poverty Monitoring&Analysis; Science Education; Scientific Research&Science Parks; Economic Theory&Research; Housing&Human Habitats;
This paper has been announced in the following NEP Reports:
- NEP-ALL-2012-03-21 (All new papers)
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Das, Jishnu & Hammer, Jeffrey & Sánchez-Paramo, Carolina, 2012.
"The impact of recall periods on reported morbidity and health seeking behavior,"
Journal of Development Economics,
Elsevier, vol. 98(1), pages 76-88.
- Jishnu Das & Jeffrey Hammer & Carolina Sánchez-Paramo, 2011. "The Impact of Recall Periods on Reported Morbidity and Health Seeking Behavior," Working Papers 1320, Princeton University, Woodrow Wilson School of Public and International Affairs, Research Program in Development Studies..
- Das, Jishnu & Hammer, Jeffrey & Sanchez-Paramo, Carolina, 2011. "The impact of recall periods on reported morbidity and health seeking behavior," Policy Research Working Paper Series 5778, The World Bank.
- Jalan, Jyotsna & Ravallion, Martin, 1998.
"Are there dynamic gains from a poor-area development program?,"
Journal of Public Economics,
Elsevier, vol. 67(1), pages 65-85, January.
- Jalan, Jyotsna & Ravallion, Martin, 1996. "Are there dynamic gains from a poor-area development program?," Policy Research Working Paper Series 1695, The World Bank.
- Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2000.
"Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score,"
NBER Technical Working Papers
0251, National Bureau of Economic Research, Inc.
- Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, 07.
- Guido Imbens, 2000. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometric Society World Congress 2000 Contributed Papers 1166, Econometric Society.
- Ravallion, Martin & Lokshin, Michael, 2000.
"Identifying welfare effects from subjective questions,"
Policy Research Working Paper Series
2301, The World Bank.
- Ravallion, Martin & Lokshin, Michael, 2001. "Identifying Welfare Effects from Subjective Questions," Economica, London School of Economics and Political Science, vol. 68(271), pages 335-57, August.
- King , Elizabeth M. & Behrman, Jere R., 2008.
"Timing and duration of exposure in evaluations of social programs,"
Policy Research Working Paper Series
4686, The World Bank.
- Elizabeth M. King & Jere R. Behrman, 2009. "Timing and Duration of Exposure in Evaluations of Social Programs," World Bank Research Observer, World Bank Group, vol. 24(1), pages 55-82, February.
- James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998.
"Characterizing Selection Bias Using Experimental Data,"
NBER Working Papers
6699, National Bureau of Economic Research, Inc.
- James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
- Heike Solga, 2001. "Longitudinal Surveys and the Study of Occupational Mobility: Panel and Retrospective Design in Comparison," Quality & Quantity: International Journal of Methodology, Springer, vol. 35(3), pages 291-309, August.
- Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
- Crump, Richard K. & Hotz, V. Joseph & Imbens, Guido W. & Mitnik, Oscar A., 2006.
"Moving the Goalposts: Addressing Limited Overlap in Estimation of Average Treatment Effects by Changing the Estimand,"
IZA Discussion Papers
2347, Institute for the Study of Labor (IZA).
- Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in Estimation of Average Treatment Effects by Changing the Estimand," Working Papers 0608, University of Miami, Department of Economics.
- Chen, Shaohua & Mu, Ren & Ravallion, Martin, 2009. "Are there lasting impacts of aid to poor areas?," Journal of Public Economics, Elsevier, vol. 93(3-4), pages 512-528, April.
- Ravallion, Martin & Chen, Shaohua, 2005. "Hidden impact? Household saving in response to a poor-area development project," Journal of Public Economics, Elsevier, vol. 89(11-12), pages 2183-2204, December.
- Anirudh Krishna & Daniel Lumonya & Milissa Markiewicz & Firminus Mugumya & Agatha Kafuko & Jonah Wegoye, 2006. "Escaping poverty and becoming poor in 36 villages of Central and Western Uganda," Journal of Development Studies, Taylor & Francis Journals, vol. 42(2), pages 346-370.
- Eva Broegaard & Ted Freeman & Carsten Schwensen, 2011. "Experience from a phased mixed-methods approach to impact evaluation of Danida support to rural transport infrastructure in Nicaragua," The Journal of Development Effectiveness, Taylor and Francis Journals, vol. 3(1), pages 9-27.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Roula I. Yazigi).
If references are entirely missing, you can add them using this form.