Judgements about loyalty: A noise audit in the context of Swedish protective security vetting

Judging an individual’s loyalty in security-sensitive roles is a high-stakes task, yet little is known about the extent and sources of variability in such judgments. This study examined how 58 participants with experience in personnel security assessment evaluated applicant profiles connected to five different countries. Each participant reviewed five protocols and judged whether the case contained information relevant to a personnel security clearance, then rated the applicant’s loyalty on a 7-point scale. Using Bayesian probit regression and an ordinal item response model with hierarchical structure, we analyzed both binary judgments and rating patterns, accounting for country of connection, applicant gender, and participant-specific variability. Results revealed substantial between-participant variability (‘noise’) in how likely judges were to flag foreign ties as relevant. Pattern noise, reflecting idiosyncratic differences in how individuals interpret the same case, was evident in loyalty ratings. Connections to Brazil and Thailand were associated with systematically lower loyalty ratings and heightened disagreement between judges, reflecting both bias and pattern noise. Contrary to policy guidance, fewer than half of foreign connections were judged as relevant, and this tendency did not vary by participant gender. The findings underscore the risk of inconsistency in high-stakes assessments and highlight the need for structured conceptual calibration in personnel vetting.