IDEAS home Printed from https://ideas.repec.org/p/wbk/wbrwps/5983.html
   My bibliography  Save this paper

Can we trust shoestring evaluations ?

Author

Listed:
  • Ravallion, Martin

Abstract

Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal,"shoestring,"method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.

Suggested Citation

  • Ravallion, Martin, 2012. "Can we trust shoestring evaluations ?," Policy Research Working Paper Series 5983, The World Bank.
  • Handle: RePEc:wbk:wbrwps:5983
    as

    Download full text from publisher

    File URL: http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2012/03/05/000158349_20120305094659/Rendered/PDF/WPS5983.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:pri:rpdevs:hammer_the_impact_of_recall_periods_on_reported_morbidity_and_health_seeking_behavior.pdf is not listed on IDEAS
    2. Elizabeth M. King & Jere R. Behrman, 2009. "Timing and Duration of Exposure in Evaluations of Social Programs," World Bank Research Observer, World Bank Group, vol. 24(1), pages 55-82, February.
    3. Chen, Shaohua & Mu, Ren & Ravallion, Martin, 2009. "Are there lasting impacts of aid to poor areas?," Journal of Public Economics, Elsevier, vol. 93(3-4), pages 512-528, April.
    4. Ravallion, Martin & Chen, Shaohua, 2005. "Hidden impact? Household saving in response to a poor-area development project," Journal of Public Economics, Elsevier, vol. 89(11-12), pages 2183-2204, December.
    5. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    6. Anirudh Krishna & Daniel Lumonya & Milissa Markiewicz & Firminus Mugumya & Agatha Kafuko & Jonah Wegoye, 2006. "Escaping poverty and becoming poor in 36 villages of Central and Western Uganda," Journal of Development Studies, Taylor & Francis Journals, vol. 42(2), pages 346-370.
    7. Heike Solga, 2001. "Longitudinal Surveys and the Study of Occupational Mobility: Panel and Retrospective Design in Comparison," Quality & Quantity: International Journal of Methodology, Springer, vol. 35(3), pages 291-309, August.
    8. Das, Jishnu & Hammer, Jeffrey & Sánchez-Paramo, Carolina, 2012. "The impact of recall periods on reported morbidity and health seeking behavior," Journal of Development Economics, Elsevier, vol. 98(1), pages 76-88.
    9. Jalan, Jyotsna & Ravallion, Martin, 1998. "Are there dynamic gains from a poor-area development program?," Journal of Public Economics, Elsevier, vol. 67(1), pages 65-85, January.
    10. Crump, Richard K. & Hotz, V. Joseph & Imbens, Guido W. & Mitnik, Oscar A., 2006. "Moving the Goalposts: Addressing Limited Overlap in Estimation of Average Treatment Effects by Changing the Estimand," IZA Discussion Papers 2347, Institute for the Study of Labor (IZA).
    11. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    12. Ravallion, Martin & Lokshin, Michael, 2001. "Identifying Welfare Effects from Subjective Questions," Economica, London School of Economics and Political Science, vol. 68(271), pages 335-357, August.
    13. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    14. Eva Broegaard & Ted Freeman & Carsten Schwensen, 2011. "Experience from a phased mixed-methods approach to impact evaluation of Danida support to rural transport infrastructure in Nicaragua," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 3(1), pages 9-27.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Minten, Bart & Singh, K.M. & Sutradhar, Rajib, 2013. "Branding and agricultural value chains in developing countries: Insights from Bihar (India)," Food Policy, Elsevier, vol. 38(C), pages 23-34.
    2. Mckenzie,David J. & Mckenzie,David J., 2016. "Can business owners form accurate counterfactuals ? eliciting treatment and control beliefs about their outcomes in the alternative treatment status," Policy Research Working Paper Series 7668, The World Bank.

    More about this item

    Keywords

    Poverty Monitoring&Analysis; Science Education; Scientific Research&Science Parks; Economic Theory&Research; Housing&Human Habitats;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wbk:wbrwps:5983. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Roula I. Yazigi). General contact details of provider: http://edirc.repec.org/data/dvewbus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.