Code like an economist: Analyzing LLMs’ code generation capabilities in economics and finance

This study investigates the code generation capabilities of large language models (LLMs) in the context of economics and finance research. Using a curated dataset of 1,088 academic papers from top-tier journals across 19 subfields, we design prompts that task a current LLM with generating code to replicate the analytical procedures described in each study. We evaluate the generated outputs for accuracy and document regeneration behaviors under autonomous (self-regeneration) and human-assisted settings. Employing a two-stage model with clustered bootstrap, we find that repeated self-regeneration improves output quality, while higher initial accuracy is associated with diminishing marginal improvements. Human-assisted regeneration further enhances code quality, particularly when built upon effective prior self-regenerations. Author expertise positively correlates with regeneration success, whereas theoretical papers are linked to lower performance, reflecting their abstract and complex nature. Robustness checks using post-2021 Q4 publications, which lie outside the LLM’s training data, confirm the consistency of these findings. Our results highlight both the promise and limitations of LLMs in supporting reproducibility and automation in computational economic and financial research.