IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v59y2025i1d10.1007_s11135-024-01950-6.html
   My bibliography  Save this article

Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with LLMs

Author

Listed:
  • Stefano De Paoli

    (Abertay University)

  • Walter S. Mathis

    (Yale School of Medicine)

Abstract

This paper presents a set of reflections on saturation and the use of Large Language Models (LLMs) for performing Thematic Analysis (TA). The paper suggests that initial thematic saturation (ITS) could be used as a metric to assess part of the transactional validity of TA with LLM, focusing on the initial coding. The paper presents the initial coding of two datasets of different sizes, and it reflects on how the LLM reaches some form of analytical saturation during the coding. The procedure proposed in this work leads to the creation of two codebooks, one comprising the total cumulative initial codes and the other the total unique codes. The paper proposes a metric to synthetically measure ITS using a simple mathematical calculation employing the ratio between slopes of the unique and total codes. The paper contributes to the initial body of work exploring how to perform qualitative analysis with LLMs.

Suggested Citation

  • Stefano De Paoli & Walter S. Mathis, 2025. "Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with LLMs," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(1), pages 683-709, February.
    • Handle: RePEc:spr:qualqt:v:59:y:2025:i:1:d:10.1007_s11135-024-01950-6
    DOI: 10.1007/s11135-024-01950-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11135-024-01950-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11135-024-01950-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Benjamin Saunders & Julius Sim & Tom Kingstone & Shula Baker & Jackie Waterfield & Bernadette Bartlam & Heather Burroughs & Clare Jinks, 2018. "Saturation in qualitative research: exploring its conceptualization and operationalization," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(4), pages 1893-1907, July.
    2. Greg Guest & Emily Namey & Mario Chen, 2020. "A simple method to assess and report thematic saturation in qualitative research," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-17, May.
    3. Hennink, Monique & Kaiser, Bonnie N., 2022. "Sample sizes for saturation in qualitative research: A systematic review of empirical tests," Social Science & Medicine, Elsevier, vol. 292(C).
    4. Roberto Franzosi & Wenqin Dong & Yilin Dong, 2022. "Qualitative and quantitative research in the humanities and social sciences: how natural language processing (NLP) can help," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(4), pages 2751-2781, August.
    5. repec:plo:pone00:0087908 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maiss Ahmad & Stephen Wilkins, 2025. "Purposive sampling in qualitative research: a framework for the entire journey," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(2), pages 1461-1479, April.
    2. Nura Abdel-Rahman & Orly Manor & Liora Valinsky & Ofri Mosenzon & Ronit Calderon-Margalit & Sveta Roberman, 2022. "What is important for people with type 2 diabetes? A focus group study to identify relevant aspects for Patient-Reported Outcome Measures in diabetes care," PLOS ONE, Public Library of Science, vol. 17(11), pages 1-14, November.
    3. Wong, Christiana & Moghimi, Ali, 2025. "Stakeholder mapping of precision weeding commercialization ecosystem in California," Agricultural Systems, Elsevier, vol. 222(C).
    4. Izudi, Jonathan & Cattamanchi, Adithya & Castelnuovo, Barbara & King, Rachel, 2024. "Barriers and facilitators to viral load suppression among people living with HIV following intensive adherence counseling in Kampala, Uganda: A qualitative study," Social Science & Medicine, Elsevier, vol. 343(C).
    5. Dario Krpan & Jonathan E. Booth & Andreea Damien, 2023. "The positive–negative–competence (PNC) model of psychological responses to representations of robots," Nature Human Behaviour, Nature, vol. 7(11), pages 1933-1954, November.
    6. Hanzhang Zhan & Bon‐Gang Hwang & Pramesh Krishnankutty, 2025. "Embracing digital transformation for sustainable development: Barriers to adopting digital twin in asset management within Singapore's energy and chemicals industry," Sustainable Development, John Wiley & Sons, Ltd., vol. 33(2), pages 2864-2887, April.
    7. Kelvin Mwita, 2022. "Factors influencing data saturation in qualitative studies," International Journal of Research in Business and Social Science (2147-4478), Center for the Strategic Studies in Business and Finance, vol. 11(4), pages 414-420, June.
    8. Nicole Brown, 2024. "Disclosure Dances in Doctoral Education," Social Sciences, MDPI, vol. 13(12), pages 1-18, December.
    9. Sanvir Sandhu & Emma Wilson & Kaushik Chattopadhyay, 2023. "Perceptions and Experiences of Undergraduate Students Regarding Social Media as a Tool for Government COVID-19-Related Messages: A Qualitative Study in Nottingham, UK," IJERPH, MDPI, vol. 20(20), pages 1-14, October.
    10. Malik, Sushant & Kajale, Dilip, 2024. "Empowering Small and Marginal Farmers: Unveiling the Potential and Addressing Obstacles of Farmer Producer Organizations in India," Research on World Agricultural Economy, Nan Yang Academy of Sciences Pte Ltd (NASS), vol. 5(01), March.
    11. Wendy Anzules-Falcones & Juan Ignacio Martin-Castilla & Ana Belén Tulcanaza-Prieto & Diana Caamana-Gomez, 2024. "Entrepreneurs in Rural Tourism: Reciprocal Influence of the Environment and Personal Factors on Business Management," Sustainability, MDPI, vol. 16(24), pages 1-19, December.
    12. Priska S. Schneider & Michelle Pantis & Christine Preiser & Daniela Hagmann & Gottfried M. Barth & Tobias J. Renner & Katharina Allgaier, 2024. "SARS-CoV-2 and Adolescent Psychiatric Emergencies at the Tübingen University Hospital: Analyzing Trends, Diagnoses, and Contributing Factors," IJERPH, MDPI, vol. 21(2), pages 1-27, February.
    13. Munjeyi Edmore & Fourie Houdini, 2024. "A Qualitative Diagnosis of Factors Influencing Presumptive Tax Collection in Zimbabwe," South Asian Journal of Business and Management Cases, , vol. 13(2), pages 166-186, August.
    14. Daniela Stelzmann & Sara Jahnke & Laura F. Kuhle, 2022. "Media Coverage of Pedophilia and Its Impact on Help-Seeking Persons with Pedophilia in Germany—A Focus Group Study," IJERPH, MDPI, vol. 19(15), pages 1-19, July.
    15. Syarah Syazwani Arifin & Izran Sarrazin Mohammad & Rohaya Abdul Jalil, 2024. "Characteristics of Effective Investment Evaluation Methods for Servicescape Intangible Value Quantification," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 8(11), pages 656-681, November.
    16. Habib, Muhammad Danish & Attri, Rekha & Salam, Mohammad Asif & Yaqub, Muhammad Zafar, 2025. "Bright and dark sides of green consumerism: An in-depth qualitative investigation in retailing context," Journal of Retailing and Consumer Services, Elsevier, vol. 82(C).
    17. Jennifer Tartaglia & Michelle McIntosh & Jonine Jancey & Jane Scott & Andrea Begley, 2021. "Exploring Feeding Practices and Food Literacy in Parents with Young Children from Disadvantaged Areas," IJERPH, MDPI, vol. 18(4), pages 1-18, February.
    18. Gülüm Özer & İdil Işık & Jordi Escartín, 2024. "Is There Somebody Looking out for Me? A Qualitative Analysis of Bullying Experiences of Individuals Diagnosed with Bipolar Disorder," IJERPH, MDPI, vol. 21(2), pages 1-22, January.
    19. Najwa Taghy & Linda Cambon & Caroline Boulliat & Olivier Aromatario & Claude Dussart, 2021. "Exploring the Determinants of Polypharmacy Prescribing and Dispensing Behaviours in Primary Care for the Elderly—Protocol for a Qualitative Study," IJERPH, MDPI, vol. 18(14), pages 1-10, July.
    20. Van Droogenbroeck, Ellen & Van Hove, Leo, 2020. "Intra-household task allocation in online grocery shopping: Together alone," Journal of Retailing and Consumer Services, Elsevier, vol. 56(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:59:y:2025:i:1:d:10.1007_s11135-024-01950-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.