IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/239.html
   My bibliography  Save this paper

Emergent Analogical Reasoning in Large Language Models: A Replication with Open-weights Alternatives

Author

Listed:
  • de Varda, Andrea Gregor
  • Saponaro, Chiara
  • Marelli, Marco

Abstract

Webb, Holyoak & Lu (2023) compared human reasoners and GPT-3 on several tasks involving analogy resolution, documenting human-level or superhuman performance in most conditions. In this direct replication, we tested a different, open-weights language model (Mixtral-8x7B) on the same materials (Experiments 1 and 2, "Digit Matrices" and "Letter String" problems) or on an augmented dataset to obtain the desired statistical power (Experiment 4, "Story Analogies"). Our replication confirmed the sign and statistical significance of the reported effects in the Digit Matrices and Story Analogies problems, whereas the model did not surpass the human baseline in the Letter String problems.

Suggested Citation

  • de Varda, Andrea Gregor & Saponaro, Chiara & Marelli, Marco, 2025. "Emergent Analogical Reasoning in Large Language Models: A Replication with Open-weights Alternatives," I4R Discussion Paper Series 239, The Institute for Replication (I4R).
    • Handle: RePEc:zbw:i4rdps:239
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/319837/1/I4R-DP239.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Abel Brodeur & Anna Dreber & Fernando Hoces de la Guardia & Edward Miguel, 2024. "Reproduction and replication at scale," Nature Human Behaviour, Nature, vol. 8(1), pages 2-3, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xia, Ziqian & Ye, Jinquan & Debnath, Ramit, 2025. "A Comment on "A Systematic Review of Worldwide Causal and Correlational Evidence on Digital Media and Democracy" by Lorenz-Spreen et al., (2023)," I4R Discussion Paper Series 206, The Institute for Replication (I4R).
    2. Mußhoff, Oliver & Ölkers, Tim & Kirchner, Ella, 2024. "A note on the discussion regarding terrorism and land use in agriculture," Land Use Policy, Elsevier, vol. 144(C).
    3. Roggenkamp, Hauke, 2025. "A comment on ‘growth and inequality in public good provision’: Testing the robustness and generalizability of dynamic public good games," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 115(C).
    4. Molo, Fabio & Pawel, Samuel & Fraga González, Gorka, 2024. "A Robustness Reproduction of "A Systematic Review and Meta-Analysis of 90 Cohort Studies of Social Isolation, Loneliness and Mortality"," I4R Discussion Paper Series 169, The Institute for Replication (I4R).
    5. Geissler, Dominique & Maarouf, Abdurahman & Bär, Dominik & Pröllochs, Nicolas & Feuerriegel, Stefan, 2025. "A comment on "A 2 million-person, campaign-wide field experiment shows how digital advertising affects voter turnout"," I4R Discussion Paper Series 237, The Institute for Replication (I4R).
    6. Bartoš, František & Godmann, Henrik R., 2025. "A Comment on "A Systematic Review and Meta-Analysis of Transdiagnostic Cognitive Behavioural Therapies for Emotional Disorders"," I4R Discussion Paper Series 213, The Institute for Replication (I4R).
    7. Siepe, Björn S. & Kloft, Matthias & Aktepe, Semih C. & Heck, Daniel W., 2025. "A Robustness Reproduction of Cox et al. (2023)," I4R Discussion Paper Series 196, The Institute for Replication (I4R).
    8. Hammar, Olle, 2025. "A Comment on "Income and Inequality in the Aztec Empire on the Eve of the Spanish Conquest"," I4R Discussion Paper Series 221, The Institute for Replication (I4R).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:239. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.