Same Question but Different Answer Experimental Evidence on Questionnaire Design's Impact on Pverty Measured by Proxies

Does the same question asked of the same population yield the same answer in face-to-face interviews when other parts of the questionnaire are altered? If not, what would be the implications for proxy-based poverty measurement? Relying on a randomized household survey experiment implemented in Malawi, this study finds that observationally equivalent as well as same households answer the same questions differently when interviewed with a short questionnaire versus the longer counterpart that, in a prior survey round, would have informed the prediction model for a proxy-based poverty measurement exercise. The analysis yields statistically significant differences in reporting between the short and long questionnaires across all topics and types of questions. The reporting differences result in significantly different predicted poverty rates and Gini coefficients. While the difference in predictions ranges from approximately 3 to 7 percentage points depending on the model specification, restricting the proxies to those collected prior the variation in questionnaire design, namely demographic variables from the household roster and location fixed effects, leads to same predictions in both samples. The findings emphasize the need for further methodological research, and suggest that short questionnaires designed for proxy-based poverty measurement should be piloted, prior to implementation, in parallel with the longer questionnaire from which they have evolved. The fact that at the median it took 25 minutes to complete the food and non-food consumption sections in the long questionnaire also implies that the implementation of these sections might not be as overly costly as usually assumed.