Hidden bias in categorical response scales: Evidence from the U.S

Social science researchers often use categorical scales (e.g., extremely unlikely, …, extremely likely) to measure behavioral intentions and assume proportional cardinality between the ordinal categories. However, if respondents have different interpretations of the categories, results will be subject to measurement error, bias, or both. This study asked respondents to associate probabilities (i.e., 0 to 100) and variability ratings (i.e., low, medium, or high) with response categories for behavioral intentions. Results show that perceived probabilities of ‘unlikely’ categories are higher than under the assumption of proportionality, and perceived probabilities of ‘likely’ categories are lower than under the assumption of proportionality. Likely categories also featured relatively higher variability. Thus, measures of behavioral changes using categorical-response scales may overestimate positive and underestimate negative intentions.