IDEAS home Printed from https://ideas.repec.org/a/spr/jcsosc/v8y2025i3d10.1007_s42001-025-00388-6.html
   My bibliography  Save this article

Prompt selection matters: enhancing text annotations for social sciences with large language models

Author

Listed:
  • Louis Abraham

    (Université Paris 1 Panthéon-Sorbonne)

  • Charles Arnal

    (Université Paris-Saclay)

  • Antoine Marie

    (Institut Jean Nicod)

Abstract

Large Language Models have recently been applied to text annotation tasks from social sciences, equating or surpassing the performance of human workers at a fraction of the cost. However, very few inquiries in the social sciences have been made of the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/ .

Suggested Citation

  • Louis Abraham & Charles Arnal & Antoine Marie, 2025. "Prompt selection matters: enhancing text annotations for social sciences with large language models," Journal of Computational Social Science, Springer, vol. 8(3), pages 1-20, August.
    • Handle: RePEc:spr:jcsosc:v:8:y:2025:i:3:d:10.1007_s42001-025-00388-6
    DOI: 10.1007/s42001-025-00388-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s42001-025-00388-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s42001-025-00388-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Joshua S. Gans & Andrew Leigh, 2012. "How Partisan is the Press? Multiple Measures of Media Slant," The Economic Record, The Economic Society of Australia, vol. 88(280), pages 127-147, March.
    2. Fabio Motoki & Valdemar Pinho Neto & Victor Rodrigues, 2024. "More human than human: measuring ChatGPT political bias," Public Choice, Springer, vol. 198(1), pages 3-23, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. George Gui & Seungwoo Kim, 2025. "Leveraging LLMs to Improve Experimental Design: A Generative Stratification Approach," Papers 2509.25709, arXiv.org.
    2. Rotaru George-Cristinel & Anagnoste Sorin & Oancea Vasile-Marian, 2024. "How Artificial Intelligence Can Influence Elections: Analyzing the Large Language Models (LLMs) Political Bias," Proceedings of the International Conference on Business Excellence, Sciendo, vol. 18(1), pages 1882-1891.
    3. Sendhil Mullainathan & Andrei Shleifer, 2005. "The Market for News," American Economic Review, American Economic Association, vol. 95(4), pages 1031-1053, September.
    4. Garz, Marcel & Szucs, Ferenc, 2023. "Algorithmic selection and supply of political news on Facebook," Information Economics and Policy, Elsevier, vol. 62(C).
    5. Olivier Toubia & George Z. Gui & Tianyi Peng & Daniel J. Merlau & Ang Li & Haozhe Chen, 2025. "Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions," Papers 2505.17479, arXiv.org.
    6. Rodrigo Taborda, 2013. "Bias in Economic News: The Reporting of Nominal Exchange Rate Behavior in Colombia," Economía Journal, The Latin American and Caribbean Economic Association - LACEA, vol. 0(Fall 2013), pages 103-153.
    7. Ryota IWAMOTO & Takunori ISHIHARA & Takanori IDA, 2025. "Comparing Risk Preferences and Loss Aversion in Humans and AI: A Persona-Based Approach with Fine-Tuning," Discussion papers e-25-006, Graduate School of Economics , Kyoto University.
    8. Hock Thye Chan, 2018. "What is the Problem Represented to Be: A Research Methodology forAnalysing Australias Skilled Migration Policy," International Journal of Business and Economic Affairs (IJBEA), Sana N. Maswadeh, vol. 3(1), pages 21-32.
    9. Jeffrey Cohen & Yuan Ding & Cédric Lesage & Hervé Stolowy, 2017. "Media Bias and the Persistence of the Expectation Gap: An Analysis of Press Articles on Corporate Fraud," Journal of Business Ethics, Springer, vol. 144(3), pages 637-659, September.
    10. König, Pascal & Rone, Julia, 2025. "From tech wars to tech culture wars? “Value shaping” and “value signaling” in AI system design and governance," SocArXiv q4xep_v1, Center for Open Science.
    11. Motoki, Fabio Y.S. & Pinho Neto, Valdemar & Rangel, Victor, 2025. "Assessing political bias and value misalignment in generative artificial intelligence," Journal of Economic Behavior & Organization, Elsevier, vol. 234(C).
    12. Yao Qu & Jue Wang, 2024. "Performance and biases of Large Language Models in public opinion simulation," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 11(1), pages 1-13, December.
    13. Tom Coupé, 2024. "Revealed Preferences: ChatGPT’s Opinion on Economic Issues and the Economics Profession," Working Papers in Economics 24/13, University of Canterbury, Department of Economics and Finance.
    14. Yingnan Yan & Tianming Liu & Yafeng Yin, 2025. "Valuing Time in Silicon: Can Large Language Model Replicate Human Value of Travel Time," Papers 2507.22244, arXiv.org.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jcsosc:v:8:y:2025:i:3:d:10.1007_s42001-025-00388-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.