Eliciting Information about Teacher Applicants Using Open-Ended Questions

Employers often use open-ended questions in hiring, but usually responses aren't recorded, and knowledge about the usefulness of such responses is limited. This study investigates properties of written responses to open-ended questions collected over many years via an application tracking platform that school districts use to hire teachers. We develop a scalable process to discover topics within the corpus of responses to each question and to score documents. It adapts a traditional qualitative topic modeling approach, Inductive Thematic Analysis, to combine the strengths of researcher judgment and large language models while preserving interpretability and transparency. Questions elicit distinct information regarding a candidate’s pedagogical skills, classroom management, professional philosophy and growth mindset, relational and cultural competence, and other professional attributes. Questions regarding classroom activities and student progress generate more dispersion in responses across applicants and fewer extreme demographic selection ratios. This framework empowers researchers and practitioners to assess language-based hiring instruments transparently and to monitor potential demographic disparities.