The Analysis of Multiple Choice Tests in Educational Assessment

Multiple choice tests are much, but not exclusively, used in the British public examinations system. The analysis of results from such tests has been subject to much debate, particularly concerning the appropriateness of latent trait models. In this paper I adopt an entirely subjectivist approach. I believe the purpose of a public examination is not to measure in some objective sense the performances of candidates, but rather to report the judgements of examiners as to those performances. It is the examiners’ judgements that are modelled by marks and grades, not something directly about the candidates themselves. Adopting this viewpoint, I make two groups of comments pertinent to multiple choice tests. First, if one is to use latent trait models to analyse candidate responses, then one must be clear as to the meaning of parameters within the models. I argue that latent trait variables are technical devices which encode certain expectations about the data, but other than that they have no physical meaning. Because of this view, I shall argue that latent trait models are appropriate for critically evaluating assumptions about examination data, but are inappropriate for the purpose of ranking candidates’ work to report and grade individual performances. Second, one should consider in what form to elicit responses from the candidates. De Finetti suggested that candidates should respond with their probability of the correctness of each possible answer to an item and that these responses should be assessed by means of a scoring rule. However, such schemes have many problems: the difficulty of getting candidates, still at school, to accept the inevitability of uncertainty in their lives; the problem of calibration, because they are unlikely to be equally good probability assessors. Perhaps more serious is the difficulty that a scoring rule which encourages a candidate to honestly reveal his beliefs may not reflect the manner in which the examiners wish to judge the candidate.