Measuring Norms and Enumerator Effects: Survey Method Matters

The reliability of quantitative data is a prerequisite for the study and design of sound public policy. However, the process of data collection and the way in which individuals are interviewed affects the data collected and can lead to bias. While this process directly impacts the quality of the data, little empirical evidence investigates the key role of the survey methods themselves. In this paper, we compare two survey methods - the standard face-to-face interview and an alternative method we call in-group individual survey. In the latter, respondents are guided by an enumerator who reads them the questions, but they answer individually and privately on an electronic device. Taking advantage of an RCT in Benin, we randomize the survey method across respondents while holding the questionnaire constant. We show that the survey method leads to different results depending on the degree of enumerator influence. Identifying this influence by quantifying how much of the variation in the outcome variable is attributable to enumerators, we document that variables that are unlikely to be influenced by enumerators do not differ significantly across survey methods. However, variables that are likely to be affected differ systematically. These variables are mainly related to norms, opinions, and beliefs. In particular, we find that respondents who answer directly on an electronic device report less gender-equal behavior and values. To rule out other mechanisms, we show that social desirability bias is more likely to affect responses in classical face-to-face interviews, where individuals' responses are less confidential.