The effects of ideology and cognitive reflection on evidence gathering behavior in the political domain

This paper explores individual-level standards of evidence in the political domain. In particular, we examine why people rely on different types of evidence in their evaluations of causal claims. Our empirical analysis is based on original survey data collected in August 2023. We conducted a demographically diverse online survey in the U.S in which we asked respondents to evaluate the effectiveness of a new policy initiative (cash bail reform). The survey offered subjects different pieces of information to evaluate the effectiveness of the intervention. Among other things, people could view: (a) The number of instances in which cities have/ have NOT been exposed to the policy intervention as well as observed societal outcomes for each case group; (b) Evaluations provided by in-group and out-group sources. Our empirical analysis reveals three major findings. First, standards of evidence vary systematically across individuals. In particular, respondents differ across two main dimensions: 1) the type of first-order/ statistical evidence they collect on a given question and 2) the type of expert testimony that they consult when assessing social cause-and-effect relationships. Second, both conservative ideology and people’s overall propensity to engage in cognitive reflection explain at least some of this variation. In particular, more liberal respondents as well as subjects with higher scores on the cognitive reflection scale exhibit a pronounced tendency to collect comprehensive statistical evidence rather than other forms of information. Third, people who score highly in cognitive refection are also more likely to refer to a broader range of external sources than their counterparts with lower reflection scores.