How Serious Are Experimenter Demand Effects? Evidence and Mitigation Strategies

Experimenter demand effects arise when participants in an experiment or survey distort their behavior in a misguided attempt to please the experimenter by confirming their research hypothesis. Experimental economists have taken the threat of demand effects seriously and have developed an array of best practices to mitigate their influence, as well as bounding techniques to assess their potential impact on inference. We provide an overview of these techniques and summarize recent empirical assessments of the potential threat of experimenter demand. Our main message is that good design is normally sufficient to address demand concerns, and that bounding approaches work well when concerns remain. Existing empirical evidence suggests that the potential impact of experimenter demand is limited.