IDEAS home Printed from https://ideas.repec.org/p/upf/upfgen/1864.html
   My bibliography  Save this paper

Uncovering the semantics of concepts using GPT-4 and Other recent large language models

Author

Listed:

Abstract

Recently, the world's attention has been captivated by Large Language Models (LLMs) thanks to OpenAI's Chat-GPT, which rapidly proliferated as an app powered by GPT-3 and now its successor, GPT-4. If these LLMs produce human-like text, the semantic spaces they construct likely align with those used by humans for interpreting and generating language. This suggests that social scientists could use these LLMs to construct measures of semantic similarity that match human judgment. In this article, we provide an empirical test of this intuition. We use GPT-4 to construct a new measure of typicality– the similarity of a text document to a concept or category. We evaluate its performance against other model-based typicality measures in terms of their correspondence with human typicality ratings. We conduct this comparative analysis in two domains: the typicality of books in literary genres (using an existing dataset of book descriptions) and the typicality of tweets authored by US Congress members in the Democratic and Republican parties (using a novel dataset). The GPT-4 Typicality measure not only meets or exceeds the current state-of-the-art but accomplishes this without any model training. This is a breakthrough because the previous state-of-the-art measure required fine-tuning a model (a BERT text classifier) on hundreds of thousands of text documents to achieve its performance. Our comparative analysis emphasizes the need for systematic empirical validation of measures based on LLMs: several measures based on other recent LLMs achieve at best a moderate correspondence with human judgments.

Suggested Citation

  • Gaël Le Mens & Balász Kovács & Michael T. Hannan & Guillem Pros, 2023. "Uncovering the semantics of concepts using GPT-4 and Other recent large language models," Economics Working Papers 1864, Department of Economics and Business, Universitat Pompeu Fabra.
  • Handle: RePEc:upf:upfgen:1864
    as

    Download full text from publisher

    File URL: https://econ-papers.upf.edu/papers/1864.pdf
    File Function: Whole Paper
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Michael T. Hannan & László Pólos & Glenn R. Carroll, 2007. "Language Matters, from Logics of Organization Theory: Audiences, Codes, and Ecologies," Introductory Chapters, in: Logics of Organization Theory: Audiences, Codes, and Ecologies, Princeton University Press.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David W. Lehman & Balázs Kovács & Glenn R. Carroll, 2014. "Conflicting Social Codes and Organizations: Hygiene and Authenticity in Consumer Evaluations of Restaurants," Management Science, INFORMS, vol. 60(10), pages 2602-2617, October.
    2. Elizabeth G. Pontikes & William P. Barnett, 2015. "The Persistence of Lenient Market Categories," Organization Science, INFORMS, vol. 26(5), pages 1415-1431, October.
    3. Xu, Jin & Peng, Biyu & Cornelissen, Joep, 2021. "Modelling the network economy: A population ecology perspective on network dynamics," Technovation, Elsevier, vol. 102(C).
    4. Rousselière, Damien & Joly, Iragäel, 2011. "A propos de la capacité à survivre des coopératives : une étude de la relation entre âge et mortalité des organisations coopératives agricoles françaises," Review of Agricultural and Environmental Studies - Revue d'Etudes en Agriculture et Environnement (RAEStud), Institut National de la Recherche Agronomique (INRA), vol. 92(3).
    5. Pierpaolo Andriani & Carsten Herrmann-Pillath, 2015. "Transactional innovation as performative action: transforming comparative advantage in the global coffee business," Journal of Evolutionary Economics, Springer, vol. 25(2), pages 371-400, April.
    6. Yu-Chieh Lo, Jade, 2015. "Selling science: Resource mobilization strategies in the emerging field of nanotechnology," Research Policy, Elsevier, vol. 44(8), pages 1513-1526.
    7. Peng, George Z. & Beamish, Paul W., 2019. "Subnational FDI Legitimacy and Foreign Subsidiary Survival," Journal of International Management, Elsevier, vol. 25(3), pages 1-1.
    8. Susan Olzak, 2022. "The Impact of Ideological Ambiguity on Terrorist Organizations," Journal of Conflict Resolution, Peace Science Society (International), vol. 66(4-5), pages 836-866, May.
    9. Robert J. David & Wesley D. Sine & Heather A. Haveman, 2013. "Seizing Opportunity in Emerging Fields: How Institutional Entrepreneurs Legitimated the Professional Form of Management Consulting," Organization Science, INFORMS, vol. 24(2), pages 356-377, April.
    10. Hedström, Peter & Wennberg, Karl, 2016. "Causal Mechanisms in Organization and Innovation Studies," Ratio Working Papers 284, The Ratio Institute.
    11. Debadutta K. Panda, 2019. "Competitive dynamics in not-for-profit organizations: evidence from India," Asia Pacific Journal of Management, Springer, vol. 36(4), pages 1251-1274, December.
    12. Michael Lounsbury & Christine M. Beckman, 2015. "Celebrating Organization Theory," Journal of Management Studies, Wiley Blackwell, vol. 52(2), pages 288-308, March.
    13. Valentina A. Assenova & Olav Sorenson, 2017. "Legitimacy and the Benefits of Firm Formalization," Organization Science, INFORMS, vol. 28(5), pages 804-818, October.
    14. Ad van den Oord & Arjen van Witteloostuijn, 2018. "A multi-level model of emerging technology: An empirical study of the evolution of biotechnology from 1976 to 2003," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-27, May.
    15. Chirag Kasbekar, 2020. "Adaptation of New Organizations to Legitimacy Shocks: Postbellum Firearms Firms in the U.S. South, 1866–1914," Organization Science, INFORMS, vol. 31(2), pages 355-377, March.
    16. Gaël Le Mens & Michael T. Hannan & László Pólos, 2015. "Organizational Obsolescence, Drifting Tastes, and Age Dependence in Organizational Life Chances," Organization Science, INFORMS, vol. 26(2), pages 550-570, April.
    17. Arzi Adbi, 2023. "Financial Sustainability of For-Profit Versus Non-Profit Microfinance Organizations Following a Scandal," Journal of Business Ethics, Springer, vol. 188(1), pages 57-74, November.
    18. Florian Überbacher, 2014. "Legitimation of New Ventures: A Review and Research Programme," Journal of Management Studies, Wiley Blackwell, vol. 51(4), pages 667-698, June.
    19. César García-Díaz & Arjen Van Witteloostuijn & Gábor Péli, 2008. "Market Dimensionality And The Proliferation Of Small-Scale Firms," Advances in Complex Systems (ACS), World Scientific Publishing Co. Pte. Ltd., vol. 11(02), pages 231-247.
    20. Giovanni Gavetti, 2012. "PERSPECTIVE—Toward a Behavioral Theory of Strategy," Organization Science, INFORMS, vol. 23(1), pages 267-285, February.

    More about this item

    Keywords

    categories; concepts; deep learning; typicality; GPT; chatGPT; BERT; Similarity;
    All these keywords.

    JEL classification:

    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:upf:upfgen:1864. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: http://www.econ.upf.edu/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.