A comment on "An Empirical Investigation of the Impact of ChatGPT on Creativity"

This report describes a computational reproduction of Lee and Chung's (2024) paper, which examined whether using ChatGPT (GPT‑3.5) enhances creativity in adults compared to web-search assistance or no assistance. The authors presented six randomized controlled experiments showing that ChatGPT-assisted responses were assessed as significantly more creative (effect sizes ranging from Cohen's d = 0.32 to 1.88). These effects were robust across diverse tasks and contexts. We first computationally reproduced all the main results using the original dataset and code, obtaining the same results as those presented by the authors in their paper. During the reproduction process, we identified two minor coding errors and one typographical error in the original table, none of which affected the substantive conclusions. Second, we performed a recreate reproduction for the main analysis in Experiments 1 and 3 by writing new R code. Our results again matched the results presented in the original paper. Overall, based on our analyses, the study is fully computationally reproducible from raw data, although only with access to the original code, due to undocumented cleaning steps, some non-described exclusion criteria, and missing codebooks. Several analyses in the original paper showed that ideas generated by ChatGPT are rated as similarly creative regardless of whether people modify them or not. We contributed to this conclusion by introducing a new robustness check using response time as a proxy for human effort in modifying ChatGPT outputs. Using data from Experiment 3, we found no significant correlation between response time and creativity in the ChatGPT condition (r = −.079, p = .449) and no moderating effect of response time on the influence of using ChatGPT on creativity. This suggests that human effort does not incrementally improve creativity beyond ChatGPT's contribution. Taken together, our findings support the original claim that using ChatGPT increases creativity regardless of the human input.