Some Conditions (Not) Affecting Selection Neglect: Evidence From The Lab

People often extrapolate from data samples, inferring properties of the population like the rate of some event, class, or group - e.g. the percent of female scientists, the crime rate, the chances to suffer some illness. In many circumstances, though, the sample observed is non-random, i.e., affected by sampling bias. For instance, news media rarely display (intentionally or not) a balanced view of the state of the world, focusing particularly on dramatic and rare events. In this line, recent literature in Economics hints that people often fail to account for sample selection in their inferences. We here offer evidence of this phenomenon at an individual level in a tightly controlled lab setting and explore conditions for its occurrence. If the inference problem is simple enough, we conjecture that the key condition is the existence of ambiguity, i.e., non-quantifiable uncertainty, about the selection rule. In this vein, we find no evidence for selection neglect in an experimental treatment, in which subjects must infer the frequency of some event given a non-random sample knowing the exact selection rule. We also consider two treatments of similar complexity where the selection rule is ambiguous. Here, in contrast, people extrapolate as if sampling were random. Further, they become more and more confident in the accuracy of their guesses as the experiment proceeds, even when the evidence accumulated patently signals a selection issue and hence warrants some caution in the inferences made. This is also true when the instructions give explicit clues about selection problems. The evidence suggests that the mere accumulation of evidence, i.e., a larger sample, will not make people more circumspect about the quality of the sample and hence about the inferences derived from it in a selection problem, even if the sample becomes obviously biased as it grows and people are reminded of the existence of potential sampling issues.