Comparable Grading from Observational Data: Many-Facet Modelling with Soft Anchors

This paper addresses grade comparability across exam cohorts when assessors and item sets change from year to year. Ordinal item scores reflect a mixture of student ability, item difficulty, and assessor severity; separating these components requires linking assumptions rarely verified empirically. We fit a sequence of Bayesian cumulative logit models to item-level scores from nine cohorts of an undergraduate statistics course. The setting is fully observational with no cross-grading and only partial assessor overlap, so cross-cohort alignment relies on repeated content used as anchors and on shared assessors. Sequential model expansion guided by posterior predictive checks reveals that treating anchors as having fixed difficulty across cohorts is inconsistent with the data. A soft-linking formulation, where linked items share a baseline difficulty but admit cohort-specific deviations regularised toward zero, removes the systematic misfit without discarding anchor information. Approximate cross-validation confirms that each modelling step improves out-of-sample predictive accuracy. Student ability estimates are robust to anchor specification (pairwise correlations exceeding 0.996), whereas cohort location estimates shift materially, which is a finding with direct consequences for grading policy. Using the recovered ability scale, we construct counterfactual grades and show that assessor severity is the dominant predictor of individual grade movement.