Program monitoring of educational tablet-based interventions using topic modeling in Stata

Rigorous research conducted in Africa since 2015 established that onebillion's software, an award-winning tablet-based curriculum, produces meaningful impacts in literacy and numeracy (Levesque, Bardack, Chigeda 2020; Levesque, Bardack, Chigeda, Bahlibi, Winiko 2022; Pitchford, Hubber, Chigeda 2017). As these programs are scaled up, program monitoring will become critical for maintaining the quality of implementation and outcomes. International organizations have called for using text analysis as a tool for monitoring and evaluation (Wencker 2019). The present study piloted the use of text analysis to identify themes from field observations of a tablet-based program using onebillion's software for early grade learners. We collected 426 open-ended observations by field officers. We used the Stata package ldagibbs to run topic modeling/latent Dirichlet allocation (LDA). LDA clusters text documents into a user-chosen number of topics (Schwarz 2018). We anticipated that LDA would generate topics that help us more efficiently summarize field observations. LDA successfully generated topics such as faulty audio cables and how they contributed to noisier classrooms. We will receive more survey data as we scale to new sites. Pilot results suggest that LDA may be an efficient means of identifying topics otherwise difficult to identify with staff review of voluminous survey responses.