IDEAS home Printed from https://ideas.repec.org/p/feb/artefa/00796.html

12 Best Practices for Leveraging Generative AI in Experimental Research

Author

Listed:
  • Samuel Chang
  • Andrew Kennedy
  • Aaron Leonard
  • John List

Abstract

We provide twelve best practices and discuss how each practice can help researchers accurately, credibly, and ethically use Generative AI (GenAI) to enhance experimental research. We split the twelve practices into four areas. First, in the pre-treatment stage, we discuss how GenAI can aid in pre-registration procedures, data privacy concerns, and ethical considerations specific to GenAI usage. Second, in the design and implementation stage, we focus on GenAI's role in identifying new channels of variation, piloting and documentation, and upholding the four exclusion restrictions. Third, in the analysis stage, we explore how prompting and training set bias can impact results as well as necessary steps to ensure replicability. Finally, we discuss forward-looking best practices that are likely to gain importance as GenAI evolves.

Suggested Citation

  • Samuel Chang & Andrew Kennedy & Aaron Leonard & John List, 2024. "12 Best Practices for Leveraging Generative AI in Experimental Research," Artefactual Field Experiments 00796, The Field Experiments Website.
    • Handle: RePEc:feb:artefa:00796
    as

    Download full text from publisher

    File URL: http://s3.amazonaws.com/fieldexperiments-papers2/papers/00796.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2025. "Generative AI at Work," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 140(2), pages 889-942.
    2. Jens Ludwig & Sendhil Mullainathan, 2024. "Machine Learning as a Tool for Hypothesis Generation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 139(2), pages 751-827.
    3. Gary Charness & Brian Jabarian & John A. List, 2023. "Generation Next: Experimentation with AI," NBER Working Papers 31679, National Bureau of Economic Research, Inc.
    4. John List, 2024. "Experimental Economics: Theory and Practice," Natural Field Experiments 00792, The Field Experiments Website.
    5. Melissa Dell, 2025. "Deep Learning for Economists," Journal of Economic Literature, American Economic Association, vol. 63(1), pages 5-58, March.
    6. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    7. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    8. John A. List, 2024. "Optimally generate policy-based evidence before scaling," Nature, Nature, vol. 626(7999), pages 491-499, February.
    9. John List, 2021. "The Voltage Effect in Behavioral Economics," Artefactual Field Experiments 00733, The Field Experiments Website.
    10. Argyle, Lisa P. & Busby, Ethan C. & Fulda, Nancy & Gubler, Joshua R. & Rytting, Christopher & Wingate, David, 2023. "Out of One, Many: Using Language Models to Simulate Human Samples," Political Analysis, Cambridge University Press, vol. 31(3), pages 337-351, July.
    11. Anton Korinek, 2024. "Generative AI for Economic Research: LLMs Learn to Collaborate and Reason," NBER Working Papers 33198, National Bureau of Economic Research, Inc.
    12. John J. Horton & Apostolos Filippas & Benjamin S. Manning, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," NBER Working Papers 31122, National Bureau of Economic Research, Inc.
    13. John J. Horton & Apostolos Filippas & Benjamin S. Manning, 2023. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?," Papers 2301.07543, arXiv.org, revised Feb 2026.
    14. Peltzman, Sam, 1975. "The Effects of Automobile Safety Regulation," Journal of Political Economy, University of Chicago Press, vol. 83(4), pages 677-725, August.
    15. repec:ces:ceswps:_10666 is not listed on IDEAS
    16. List, John A, et al, 2001. "Academic Economists Behaving Badly? A Survey on Three Areas of Unethical Behavior," Economic Inquiry, Western Economic Association International, vol. 39(1), pages 162-170, January.
    17. John A. List & Ian Muir & Gregory Sun, 2024. "Using machine learning for efficient flexible regression adjustment in economic experiments," Econometric Reviews, Taylor & Francis Journals, vol. 44(1), pages 2-40, July.
    18. Felix Chopra & Ingar Haaland, 2023. "Conducting qualitative interviews with AI," CEBI working paper series 23-06, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ji Ma, 2025. "Computational Basis of LLM's Decision Making in Social Simulation," Papers 2504.11671, arXiv.org, revised Dec 2025.
    2. Zarifhonarvar, Ali, 2026. "Generating inflation expectations with large language models," Journal of Monetary Economics, Elsevier, vol. 157(C).
    3. Shu Wang & Zijun Yao & Shuhuai Zhang & Jianuo Gai & Tracy Xiao Liu & Songfa Zhong, 2025. "When Experimental Economics Meets Large Language Models: Evidence-based Tactics," Papers 2505.21371, arXiv.org, revised Jul 2025.
    4. Keloharju, Matti & Keluharju, Roope, 2025. "Accounting Research in the Age of AI," Working Paper Series 1528, Research Institute of Industrial Economics, revised 29 Jul 2025.
    5. Kiwoong Yoo & Michael Haenlein & Kelly Hewett, 2025. "A whole new world, a new fantastic point of view: Charting unexplored territories in consumer research with generative artificial intelligence," Journal of the Academy of Marketing Science, Springer, vol. 53(3), pages 723-759, May.
    6. Ma, Ji, 2025. "Steering Prosocial AI Agents: Computational Basis of LLM's Decision Making in Social Simulation," OSF Preprints 8p7wg_v1, Center for Open Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rosa-García, Alfonso, 2024. "Student Reactions to AI-Replicant Professor in an Econ101 Teaching Video," MPRA Paper 120135, University Library of Munich, Germany.
    2. John A. List, 2024. "Optimally generate policy-based evidence before scaling," Nature, Nature, vol. 626(7999), pages 491-499, February.
    3. repec:osf:osfxxx:r3qng_v1 is not listed on IDEAS
    4. Paola Cillo & Gaia Rubera, 2025. "Generative AI in innovation and marketing processes: A roadmap of research opportunities," Journal of the Academy of Marketing Science, Springer, vol. 53(3), pages 684-701, May.
    5. Matthew O. Jackson & Qiaozhu Me & Stephanie W. Wang & Yutong Xie & Walter Yuan & Seth Benzell & Erik Brynjolfsson & Colin F. Camerer & James Evans & Brian Jabarian & Jon Kleinberg & Juanjuan Meng & Se, 2025. "AI Behavioral Science: A Framework and Agenda," Papers 2509.13323, arXiv.org, revised Oct 2026.
    6. Guillaume Coqueret & Joan Llull & Florian Oswald & Christophe Pérignon & Christoph Scheuch & Lars Vilhuber, 2026. "Randomness In Large Language Models: What Researchers Need to Know (And Report)," Working Papers hal-05730402, HAL.
    7. Nikoleta Anesti & Edward Hill & Andreas Joseph, 2025. "Inflation Attitudes of Large Language Models," Papers 2512.14306, arXiv.org.
    8. repec:ces:ceswps:_10666 is not listed on IDEAS
    9. Hua Li & Qifang Wang & Ye Wu, 2025. "From Mobile Media to Generative AI: The Evolutionary Logic of Computational Social Science Across Data, Methods, and Theory," Mathematics, MDPI, vol. 13(19), pages 1-17, September.
    10. Feyzollahi, Maryam & Rafizadeh, Nima, 2025. "The adoption of Large Language Models in economics research," Economics Letters, Elsevier, vol. 250(C).
    11. Ben Weidmann & Yixian Xu & David J. Deming, 2025. "Measuring Human Leadership Skills with Artificially Intelligent Agents," Papers 2508.02966, arXiv.org.
    12. Wenjun Cao, 2025. "Black Box Absorption: LLMs Undermining Innovative Ideas," Papers 2510.20612, arXiv.org.
    13. Iñaki Aldasoro & Ajit Desai, 2025. "Money Talks: AI Agents for Cash Management in Payment Systems," Staff Working Papers 25-35, Bank of Canada.
    14. Felipe A. Csaszar & Harsh Ketkar & Hyunjin Kim, 2024. "Artificial Intelligence and Strategic Decision-Making: Evidence from Entrepreneurs and Investors," Strategy Science, INFORMS, vol. 9(4), pages 322-345, December.
    15. Shumiao Ouyang & Hayong Yun & Xingjian Zheng, 2024. "AI as Decision-Maker: Ethics and Risk Preferences of LLMs," Papers 2406.01168, arXiv.org, revised Jun 2025.
    16. John R. Graham & Campbell R. Harvey & Manish Jha, 2026. "CFOs Meet LLMs," Papers 2606.13812, arXiv.org, revised Sep 2026.
    17. Navid Ghaffarzadegan & Aritra Majumdar & Ross Williams & Niyousha Hosseinichimeh, 2024. "Generative agent‐based modeling: an introduction and tutorial," System Dynamics Review, System Dynamics Society, vol. 40(1), January.
    18. Hongshen Sun & Juanjuan Zhang, 2025. "From Model Choice to Model Belief: Establishing a New Measure for LLM-Based Research," Papers 2512.23184, arXiv.org.
    19. Giuseppe Matera, 2025. "Corporate Earnings Calls and Analyst Beliefs," Papers 2511.15214, arXiv.org, revised Nov 2025.
    20. Iñaki Aldasoro & Ajit Desai, 2025. "AI agents for cash management in payment systems," BIS Working Papers 1310, Bank for International Settlements.
    21. Seung Jung Lee & Anne Lundgaard Hansen, 2025. "Financial Stability Implications of Generative AI: Taming the Animal Spirits," Finance and Economics Discussion Series 2025-090, Board of Governors of the Federal Reserve System (U.S.).
    22. Koji Takahashi & Joon Suk Park, 2025. "Generative AI for Surveys on Payment Apps: AIs' View on Privacy and Technology," IMES Discussion Paper Series 25-E-13, Institute for Monetary and Economic Studies, Bank of Japan.

    More about this item

    JEL classification:

    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:feb:artefa:00796. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Franco Daniel Albino (email available below). General contact details of provider: http://www.fieldexperiments.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.