Are We Doing More Harm than Good? Using Hypothetical Bias Reduction Techniques in Consequential Decision Settings

Researchers deploying stated preference surveys to value public goods commonly use techniques designed to reduce bias in hypothetical choice settings. This practice is at odds with evidence that most survey respondents perceive that their decisions are not hypothetical but instead have economic consequences. We examine three bias reduction procedures in both hypothetical and incentive-compatible, real payment settings: cheap talk, solemn oath, and certainty adjustment. We confirm that adjusting hypothetical choices based on response certainty or using a solemn oath can reduce hypothetical bias. In the incentive-compatible decision setting, the oath increases willingness to pay (WTP), and certainty adjustment can lead to serious distortions in demand estimates. Cheap talk does not alter mean WTP but leads to a stark difference in WTP across sexes. To minimize unintended consequences, our results suggest that survey researchers should deploy screening questions to better target these hypothetical bias reduction techniques.