Evaluating Evaluations: Assessing the Quality of Aid Agency Evaluations in Global Health - Working Paper 461

Evaluations are key to learning and accountability yet the quality of those evaluations are critical to their usefulness. We assessed the methodological quality of global health program evaluations commissioned or conducted by five major funders and published between 2009 and 2014. From a universe of 299 large-scale global health program evaluations, we randomly selected 37 evaluations stratified by whether they were performance evaluations or impact evaluations and applied a systematic assessment approach with two reviewers scoring each evaluation. We found that most evaluations did not meet social science methodological standards in terms of using methods and data that would simultaneously assure relevance, validity, and reliability. Most evaluations (76 percent) asked questions relevant to the health program, but 43 percent of evaluations failed to collect relevant data. In addition, only about a fifth of the evaluations followed accepted social science methods for sampling. We also assessed whether evaluations took a systematic analytical approach and considered potential confounding variables. In this regard, only 16 percent of evaluations in our sample had high analytical validity and reliability. The study provides ten recommendations for improving the quality of evaluations, including a robust finding that early planning of evaluations is associated with better quality and noting the value of better sampling approaches in data collection and disclosure of potential conflicts of interest and data.