Assessing Model Sensitivity of the Imputation Methods Used in the National Assessment of Educational Progress

The National Assessment of Educational Progress (NAEP) uses latent trait item response models to summarize performance of students on assessments of educational proficiency in different subject areas such as mathematics and reading. Because of limited examination time and concerns about student motivation. NAEP employs sparse matrix sampling designs that assign a small number of examination items to each sampled student to measure broad curriculums. As a consequence, each sampled student’s latent trait is not accurately measured, and NAEP uses multiple imputation missing data statistical methods to account for the uncertainty about the latent traits. The sensitivity of these model-based estimation and reporting procedures to statistical and psychometric assumptions is assessed. Estimation of the mean of the latent trait train different subpopulations was very robust to the modeling assumptions. Many of the other currently reported summaries, however; may depend on the modeling assumptions underlying the estimation procedures; these assumptions, motivated primarily by analytic tractability, are unlikely to attain, raising concerns about current reporting practices. The results indicate that more conservative criteria should be considered when forming intervals about estimates, and when assessing significance. A possible expansion of the imputation model is suggested that may improve its performance.