Large Language Models for Statistical Analysis: Can they Replace Domain-Specific Software Packages?

Large language models (LLMs) represent one of the most significant advances in artificial intelligence in recent decades. Although LLMs are widely used to generate statistical code in domain-specific languages such as R, Python, and SAS, they are increasingly employed to perform data analysis directly. Prior studies have examined the use of LLMs in statistical analysis, but important gaps remain, including limited evidence on their proficiency in data manipulation and Bayesian statistical modeling. The present study was designed to evaluate the performance of common LLMs across a wide range of data analysis tasks, including data reading, data manipulation, descriptive statistics, contingency table analysis, mean comparison tests, correlation analysis, regression modeling, and Bayesian inference. Six large language models were assessed: ChatGPT 5.3, Gemini 3.1, Claude Sonnet 4.6, Microsoft Copilot GPT 5.1, Grok 4.2, and DeepSeek 3.2. All models were tested using their free-tier access, except for ChatGPT, which was evaluated through a paid subscription. Fully or partially simulated datasets were used in this study, and strict scoring criteria were implemented, in that the outputs from the LLMs must be consistent with those from R, and they must be reproducible upon re-run. Gemini, ChatGPT, and Claude achieved 100% accuracy in data reading and descriptive statistics. Gemini and Claude generated correct results for mean comparison tests. ChatGPT and Claude produced accurate outputs in correlation and regression analyses. None of the LLMs achieved 100% accuracy in data manipulation, contingency table analyses, and Bayesian modeling. On average, no LLM achieved perfect accuracy. The overall performance of Gemini, ChatGPT, and Claude was comparable, whereas Grok, Copilot, and DeepSeek performed poorly. Limitations in data manipulation and some inferential statistical methods suggest that LLMs cannot yet replace domain-specific software packages. Therefore, LLMs are better suited as complementary tools rather than standalone applications for rigorous statistical analysis.