Automating Evidence Synthesis: A Comparative Evaluation of Large Language Models for Data Extraction

Systematic reviews and meta-analyses (SRMAs) are important tools for evidence synthesis but have historically required substantial manual effort, particularly during the data extraction phase. To address this bottleneck, we developed and evaluated an automated pipeline that utilizes large language models (LLMs) to ingest full text scientific articles and extract structured metadata. We benchmarked the performance of leading models, including Gemini 2.5 Pro, GPT-5, and Sonnet 4.0, across two distinct domains: mobile health interventions and education. Our results indicate that Gemini 2.5 Pro achieved the strongest performance in qualitative metadata extraction and outcome identification. However, quantitative metadata extraction remained a significant challenge. Models struggled to interpret complex data across multiple tables and failed to calculate effect sizes when only raw figures were reported. Crucially, we find that human annotators often applied implicit filtering criteria not documented in the coding manual, which made benchmarking the results challenging. We discuss the implications of these findings, emphasizing that while LLMs can accelerate the coding process, reliable automation requires significantly more prescriptive coding manuals to strictly steer model behavior and ensure fair benchmarking.