Bias from the use of mean-based methods on test scores

Economists regularly regress IQ scores or achievement test scores on covariates, for example to evaluate educational policy. These test scores are ordinal measures, and their distributions can take an arbitrary shape, even though they are often constructed to look normal. The ordinality of test scores makes the use of mean-based methods such as OLS is inappropriate: estimates are not robust to changes in test score estimation assumptions and methods. I simulate the magnitude of robustness problems, and show that in practice, problems with mean-based regression of normally distributed test scores are small. Even so, test score distributions with more exotic shapes will need to be transformed before use.