Validating an AI Grader for Community Corrections Supervision Visits: A Reliability and Agreement Study Against Expert Human Auditors

Meta-analytic evidence links sustained officer fidelity to core correctional practices with substantially lower participant recidivism; caseloads supervised by untrained officers recidivate roughly 39% more often than those supervised by trained officers (Chadwick et al., 2015). That fidelity, however, can be measured today only by having trained humans hand-code recordings, which does not scale beyond a small audit sample. We test whether a frontier large language model can grade recorded supervision visits against a rubric of observable, standards-based officer behaviors as reliably as expert human auditors. On a 30-visit corpus drawn from multiple agencies and independently scored by three expert graders, we report a descriptive validation of an LLM grader (Claude Sonnet 5 running a calibrated prompt, scored as a three-run ensemble). The expert graders agree only moderately with one another (item-level agreement 65–73%, Gwet’s AC1 +0.34 to +0.46; weighted-total Krippendorff’s α +0.199), a level typical of subjective behavioral coding. Against this panel, the model lands at least as close to the panel average as the median human grader on 23 of 30 visits (77%, 95% CI 0.60–0.90; single-run range 20–24 of 30; 16 of 18 on the never-seen holdout). It agrees with the panel at the item level about as well as the graders agree with each other (macro-F1 0.57–0.74), sits closer to each grader than the graders sit to one another, and, consistent with sitting near the panel’s center, raises the panel’s agreement statistic when added as a member. Grader-prompt performance is tied to the specific model version. We scope the tool to officer coaching, not personnel decisions or participant outcomes, and discuss deployment, reliability, cost, and limitations, along with next steps: an adjudication study to establish a consensus reference, and a study of whether the model can coach as well as a human.