Expert Validation of a Qualitative Interview Protocol for ESL Instructors’ Digital Assessment Literacy in Malaysian Higher Education: A Delphi Study

The integration of technology into educational assessment is a critical component of modern pedagogy, yet many educators lack proficiency in digital assessment methodologies. This study addresses the need for a validated instrument to explore the digital assessment literacy (DAL) of English as a Second Language (ESL) instructors in the Malaysian higher education context. The primary objective was to validate a qualitative interview protocol designed to investigate instructors’ assessment practices and their level of technology integration. The protocol consists of two instruments, one guided by the Teacher Assessment Literacy in Practice (TALiP) framework and the other by the Substitution, Augmentation, Modification, and Redefinition (SAMR) model. A Delphi method was employed to achieve consensus on the instrument’s content validity. Six experts in the fields of digital assessment and ESL, holding PhDs with over five years of relevant experience, were selected to evaluate the interview items. Inter-rater reliability was calculated using the Fleiss’ Kappa statistic. The results indicated an “almost perfect†level of agreement among the expert validators, with a Kappa value of 0.81 for the TALiP instrument and 0.83 for the SAMR instrument. Based on expert feedback, several items were refined for clarity, including the separation of double-barreled questions and the rephrasing of complex terminology, which increased the total number of items in the TALiP protocol from 10 to 15. The study concludes that the validated interview protocol is a robust and reliable instrument for exploring the multifaceted dimensions of DAL among ESL instructors. This tool can be effectively utilized by researchers and educational institutions to inform the design of targeted professional development programs aimed at enhancing digital assessment practices in higher education.