Evaluating AI in Higher Education: Metrics, Evidence Quality, and Meta-Analytic Synthesis

Applied computing and artificial intelligence (AI) are increasingly implemented in higher education, yet evidence about their “impact” remains difficult to compare because studies use heterogeneous outcome metrics and inconsistent reporting. This study conducts a PRISMA 2020–guided systematic review and meta-analysis to evaluate how impact is measured in empirical research on AI/applied computing in higher education and to synthesize effects when data permit. Searches were performed in IEEE Xplore, ACM Digital Library, Scopus, SpringerLink, and Google Scholar (2020–2024). From 1,610 records, eight studies met inclusion criteria. Outcomes were reported as learning outcomes/engagement and technical AI model performance metrics, revealing substantial variability in operational definitions, instruments, and reporting completeness. Using a random-effects model, the pooled effect for learning outcomes and engagement was SMD = 3.32 (95% CI [2.77, 3.87]) with extreme heterogeneity (I² ≈ 99%), indicating limited comparability across studies. The findings suggest that the main barrier to cumulative evidence is weak metric coherence rather than a lack of measurable outcomes. Future work should separate learning outcomes from engagement, avoid treating technical model metrics as educational impact unless explicitly linked to student-level outcomes, and adopt minimum reporting standards to enable robust evaluation and synthesis.