IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v54y2025i3p1197-1242.html

Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization

Author

Listed:
  • Simone Zhang
  • Janet Xu
  • AJ Alvero

Abstract

The growing popularity of generative artificial intelligence (AI) tools presents new challenges for data quality in online surveys and experiments. This study examines participants’ use of large language models to answer open-ended survey questions and describes empirical tendencies in human versus large language model (LLM)-generated text responses. In an original survey of research participants recruited from a popular online platform for sourcing social science research subjects, 34 percent reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants’ LLM use in online research.

Suggested Citation

  • Simone Zhang & Janet Xu & AJ Alvero, 2025. "Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization," Sociological Methods & Research, , vol. 54(3), pages 1197-1242, August.
    • Handle: RePEc:sae:somere:v:54:y:2025:i:3:p:1197-1242
    DOI: 10.1177/00491241251327130
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/00491241251327130
    Download Restriction: no
    File URL: https://libkey.io/10.1177/00491241251327130?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Adam J. Berinsky & Michele F. Margolis & Michael W. Sances, 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self‐Administered Surveys," American Journal of Political Science, John Wiley & Sons, vol. 58(3), pages 739-753, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David Almog & Lucas Lippman & Daniel Martin, 2026. "When an AI Judges Your Work: The Hidden Costs of Algorithmic Assessment," Papers 2603.02076, arXiv.org.
    2. Can Celebi & Christine Exley & Soren Harrs & Hannu Kivimaki & Marta Serra-Garcia & Jeffrey Yusof, 2026. "Mission Possible: The Collection of High-Quality Data," CESifo Working Paper Series 12535, CESifo.
    3. Westland, J. Christopher, 2026. "Building Lovemarks with social media marketing: A methodological comparison," Journal of Retailing and Consumer Services, Elsevier, vol. 88(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schiff, Kaylyn Jackson & Schiff, Daniel S. & Bueno, Natalia, 2023. "The Liar's Dividend: The Impact of Deepfakes and Fake News on Trust in Political Discourse," SocArXiv x43ph, Center for Open Science.
    2. Vecchio, Riccardo & Caso, Gerarda & Cembalo, Luigi & Borrello, Massimiliano, . "Is respondents’ inattention in online surveys a major issue for research?," Economia agro-alimentare / Food Economy, Italian Society of Agri-food Economics/Società Italiana di Economia Agro-Alimentare (SIEA), vol. 22(01).
    3. Lude, Maximilian & Prügl, Reinhard, 2021. "Experimental studies in family business research," Journal of Family Business Strategy, Elsevier, vol. 12(1).
    4. Simon, Mark & Stanton, Steven J. & Townsend, Janell D. & Kim, John, 2019. "A multi-method study of social ties and crowdfunding success: Opening the black box to get the cash inside," Journal of Business Research, Elsevier, vol. 104(C), pages 206-214.
    5. Francis Wong, 2024. "Taxing Homeowners Who Won’t Borrow," CESifo Working Paper Series 11185, CESifo.
    6. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    7. Jimin Pyo & Michael G. Maxfield, 2021. "Cognitive Effects of Inattentive Responding in an MTurk Sample," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2020-2039, July.
    8. Henkel, Aljosha & Fehr, Ernst & Senn, Julien & Epper, Thomas, 2025. "Beliefs about inequality and the nature of support for redistribution," Journal of Public Economics, Elsevier, vol. 246(C).
    9. Le Maux, Benoît & Necker, Sarah, 2023. "Honesty nudges: Effect varies with content but not with timing," Journal of Economic Behavior & Organization, Elsevier, vol. 207(C), pages 433-456.
    10. Schilke, Oliver & Reimann, Martin, 2025. "The transparency dilemma: How AI disclosure erodes trust," Organizational Behavior and Human Decision Processes, Elsevier, vol. 188(C).
    11. Lilith Burgstaller & Annabelle Doerr & Sarah Necker, 2023. "Do Household Tax Credits Increase the Demand for Legally Provided Services?," CESifo Working Paper Series 10211, CESifo.
    12. Anthony Fowler & Kisoo Kim, 2022. "An information-based explanation for partisan media sorting," Journal of Theoretical Politics, , vol. 34(4), pages 499-526, October.
    13. Burgstaller, Lilith & Doerr, Annabelle & Necker, Sarah, 2023. "Incentives for Consumers to Act as Tax Auditors: (When) Are They Effective?," VfS Annual Conference 2023 (Regensburg): Growth and the "sociale Frage" 277628, Verein für Socialpolitik / German Economic Association.
    14. Austin Strange, 2025. "Influence and support for foreign aid: Evidence from the United States and China," The Review of International Organizations, Springer, vol. 20(3), pages 443-470, September.
    15. Bharadwaj, Preethika & Innocenti, Stefania, 2026. "Securing public support for fuel subsidy reform: Experimental and policymaker insights from Malaysia," Energy Economics, Elsevier, vol. 153(C).
    16. Alessandro Nai, 2020. "The Trump Paradox: How Cues from a Disliked Source Foster Resistance to Persuasion," Politics and Governance, Cogitatio Press, vol. 8(1), pages 122-132.
    17. Paape, Vanessa Isabel, 2022. "The Hidden Influence of Cognitive Processing Style on Consumers' Intention to Adopt Innovative Products," Junior Management Science (JUMS), Junior Management Science e. V., vol. 7(1), pages 112-133.
    18. Milton Mayfield & Jacqueline Mayfield, 2021. "Sound and Safe: The Role of Leader Motivating Language and Follower Self-Leadership in Feelings of Psychological Safety," Administrative Sciences, MDPI, vol. 11(2), pages 1-30, May.
    19. Stephen Dansky & B. Andrew Cudmore & Amitabh S. Dutta, 2025. "Rethinking Internalization Theory due to Coase’s “Hidden in Plain Sight†Assumption," SAGE Open, , vol. 15(2), pages 21582440251, June.
    20. Branden B. Johnson & Adam M. Finkel, 2023. "Sensitivity to scope in estimating the social benefits of prolonging lives for regulatory decisions using national stated preference tradeoffs," Environment Systems and Decisions, Springer, vol. 43(3), pages 509-528, September.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:54:y:2025:i:3:p:1197-1242. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.