How to conduct impact evaluations in humanitarian and conflict settings

Methodological, ethical and practical challenges make it difficult to use experimental and rigorous quasi-experimental approaches to conduct impact evaluations in humanitarian emergencies and conflict settings (HECS). This paper discusses recent developments in the design, measurement, data and analysis of impact evaluations that can overcome these challenges and provide concrete examples from our recent research where we analyse the impact of agricultural emergency interventions in post-war Syria. More specifically, the paper offers solutions: First, discuss the challenges in designing rapid and rigorous impact evaluations in HECS. By doing so, we mainly show alternative ways to construct counterfactuals in the absence of meaningful control groups; Second, we review how researchers can use additional data sources to create a counterfactual or even data on treated units when it is difficult to collect data and in some cases provide ethical and methodological benefits in addition to providing cost-effectiveness. Third, we argue that finding and fine-tuning proxy measures for the ‘unmeasurable’ concepts and outcomes such as resilience and fragility are crucial. Fourth, we highlight how adaptive machine learning algorithms are helpful in rigorous impact evaluations in HECS to overcome the drawbacks related to data availability and heterogeneity analysis. We provide an example from our recent work where we use honest causal forest estimation to test the heterogeneous impact of an agricultural intervention when sample sizes are small. Fifth, we discuss how standardisation across methods, data and measures ensures the external validity and transferability of the evidence to other complex settings where impact evaluation is challenging to conduct. Finally, the paper recommends how future research and policy can adapt these tools to ensure significant and effective learning in conflict-affected and humanitarian settings.