LLM measurements that agree with human coders overall can still distort specific effects

Large language models (LLMs) are increasingly adopted in the social sciences as surrogates for human coders. Standard practice validates them by their correspondence with human measurements of the same stimuli using statistics like correlation. However, overall correspondence can conceal biases that distort specific claims. We demonstrate this with open-weight LLM ratings of 568 food and 1,811 scene photographs. Ratings inflate, attenuate, or reverse effects, and improving overall correspondence with prompts can improve one contrast while worsening another. Rankings also elevate some categories and demote others beyond what random errors of the same size typically produce. Across six prompts, some of these errors are shared, so sampling across prompts may continue to mask biases. We propose validating LLM coders against specific claims they will test rather than the general coding task.