IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v59y2025i1d10.1007_s11135-024-01950-6.html
   My bibliography  Save this article

Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with LLMs

Author

Listed:
  • Stefano De Paoli

    (Abertay University)

  • Walter S. Mathis

    (Yale School of Medicine)

Abstract

This paper presents a set of reflections on saturation and the use of Large Language Models (LLMs) for performing Thematic Analysis (TA). The paper suggests that initial thematic saturation (ITS) could be used as a metric to assess part of the transactional validity of TA with LLM, focusing on the initial coding. The paper presents the initial coding of two datasets of different sizes, and it reflects on how the LLM reaches some form of analytical saturation during the coding. The procedure proposed in this work leads to the creation of two codebooks, one comprising the total cumulative initial codes and the other the total unique codes. The paper proposes a metric to synthetically measure ITS using a simple mathematical calculation employing the ratio between slopes of the unique and total codes. The paper contributes to the initial body of work exploring how to perform qualitative analysis with LLMs.

Suggested Citation

  • Stefano De Paoli & Walter S. Mathis, 2025. "Reflections on inductive thematic saturation as a potential metric for measuring the validity of an inductive thematic analysis with LLMs," Quality & Quantity: International Journal of Methodology, Springer, vol. 59(1), pages 683-709, February.
    • Handle: RePEc:spr:qualqt:v:59:y:2025:i:1:d:10.1007_s11135-024-01950-6
    DOI: 10.1007/s11135-024-01950-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11135-024-01950-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11135-024-01950-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hennink, Monique & Kaiser, Bonnie N., 2022. "Sample sizes for saturation in qualitative research: A systematic review of empirical tests," Social Science & Medicine, Elsevier, vol. 292(C).
    2. Benjamin Saunders & Julius Sim & Tom Kingstone & Shula Baker & Jackie Waterfield & Bernadette Bartlam & Heather Burroughs & Clare Jinks, 2018. "Saturation in qualitative research: exploring its conceptualization and operationalization," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(4), pages 1893-1907, July.
    3. Greg Guest & Emily Namey & Mario Chen, 2020. "A simple method to assess and report thematic saturation in qualitative research," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-17, May.
    4. Roberto Franzosi & Wenqin Dong & Yilin Dong, 2022. "Qualitative and quantitative research in the humanities and social sciences: how natural language processing (NLP) can help," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(4), pages 2751-2781, August.
    5. repec:plo:pone00:0087908 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dario Krpan & Jonathan E. Booth & Andreea Damien, 2023. "The positive–negative–competence (PNC) model of psychological responses to representations of robots," Nature Human Behaviour, Nature, vol. 7(11), pages 1933-1954, November.
    2. Hanzhang Zhan & Bon‐Gang Hwang & Pramesh Krishnankutty, 2025. "Embracing digital transformation for sustainable development: Barriers to adopting digital twin in asset management within Singapore's energy and chemicals industry," Sustainable Development, John Wiley & Sons, Ltd., vol. 33(2), pages 2864-2887, April.
    3. Nicole Brown, 2024. "Disclosure Dances in Doctoral Education," Social Sciences, MDPI, vol. 13(12), pages 1-18, December.
    4. Malik, Sushant & Kajale, Dilip, 2024. "Empowering Small and Marginal Farmers: Unveiling the Potential and Addressing Obstacles of Farmer Producer Organizations in India," Research on World Agricultural Economy, Nan Yang Academy of Sciences Pte Ltd (NASS), vol. 5(01), March.
    5. Wendy Anzules-Falcones & Juan Ignacio Martin-Castilla & Ana Belén Tulcanaza-Prieto & Diana Caamana-Gomez, 2024. "Entrepreneurs in Rural Tourism: Reciprocal Influence of the Environment and Personal Factors on Business Management," Sustainability, MDPI, vol. 16(24), pages 1-19, December.
    6. Priska S. Schneider & Michelle Pantis & Christine Preiser & Daniela Hagmann & Gottfried M. Barth & Tobias J. Renner & Katharina Allgaier, 2024. "SARS-CoV-2 and Adolescent Psychiatric Emergencies at the Tübingen University Hospital: Analyzing Trends, Diagnoses, and Contributing Factors," IJERPH, MDPI, vol. 21(2), pages 1-27, February.
    7. Munjeyi Edmore & Fourie Houdini, 2024. "A Qualitative Diagnosis of Factors Influencing Presumptive Tax Collection in Zimbabwe," South Asian Journal of Business and Management Cases, , vol. 13(2), pages 166-186, August.
    8. Habib, Muhammad Danish & Attri, Rekha & Salam, Mohammad Asif & Yaqub, Muhammad Zafar, 2025. "Bright and dark sides of green consumerism: An in-depth qualitative investigation in retailing context," Journal of Retailing and Consumer Services, Elsevier, vol. 82(C).
    9. Wong, Christiana & Moghimi, Ali, 2025. "Stakeholder mapping of precision weeding commercialization ecosystem in California," Agricultural Systems, Elsevier, vol. 222(C).
    10. Izudi, Jonathan & Cattamanchi, Adithya & Castelnuovo, Barbara & King, Rachel, 2024. "Barriers and facilitators to viral load suppression among people living with HIV following intensive adherence counseling in Kampala, Uganda: A qualitative study," Social Science & Medicine, Elsevier, vol. 343(C).
    11. Kelvin Mwita, 2022. "Factors influencing data saturation in qualitative studies," International Journal of Research in Business and Social Science (2147-4478), Center for the Strategic Studies in Business and Finance, vol. 11(4), pages 414-420, June.
    12. Sanvir Sandhu & Emma Wilson & Kaushik Chattopadhyay, 2023. "Perceptions and Experiences of Undergraduate Students Regarding Social Media as a Tool for Government COVID-19-Related Messages: A Qualitative Study in Nottingham, UK," IJERPH, MDPI, vol. 20(20), pages 1-14, October.
    13. Daniela Stelzmann & Sara Jahnke & Laura F. Kuhle, 2022. "Media Coverage of Pedophilia and Its Impact on Help-Seeking Persons with Pedophilia in Germany—A Focus Group Study," IJERPH, MDPI, vol. 19(15), pages 1-19, July.
    14. Syarah Syazwani Arifin & Izran Sarrazin Mohammad & Rohaya Abdul Jalil, 2024. "Characteristics of Effective Investment Evaluation Methods for Servicescape Intangible Value Quantification," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 8(11), pages 656-681, November.
    15. Jennifer Tartaglia & Michelle McIntosh & Jonine Jancey & Jane Scott & Andrea Begley, 2021. "Exploring Feeding Practices and Food Literacy in Parents with Young Children from Disadvantaged Areas," IJERPH, MDPI, vol. 18(4), pages 1-18, February.
    16. Gülüm Özer & İdil Işık & Jordi Escartín, 2024. "Is There Somebody Looking out for Me? A Qualitative Analysis of Bullying Experiences of Individuals Diagnosed with Bipolar Disorder," IJERPH, MDPI, vol. 21(2), pages 1-22, January.
    17. Şahika Simsek-Cetinkaya & Simge Evrenol Ocal, 2023. "“Psychological Injuries Are Not Visible†: Experiences and Perceptions of Midwives and Nurses about Domestic Violence during Pregnancy," Clinical Nursing Research, , vol. 32(8), pages 1115-1123, November.
    18. Júlio Belo Fernandes & Diana Vareta & Sónia Fernandes & Ana Silva Almeida & Dina Peças & Noélia Ferreira & Liliana Roldão, 2022. "Rehabilitation Workforce Challenges to Implement Person-Centered Care," IJERPH, MDPI, vol. 19(6), pages 1-9, March.
    19. Chikako Ishizuka & Kei Aoki, 2024. "Drivers of sustained brand engagement: cases of long-term customers of hedonic and utilitarian brands in Japan," Journal of Marketing Analytics, Palgrave Macmillan, vol. 12(4), pages 979-989, December.
    20. Silvia Scaramuzzi & Sara Gabellini & Giovanni Belletti & Andrea Marescotti, 2021. "Agrobiodiversity-Oriented Food Systems between Public Policies and Private Action: A Socio-Ecological Model for Sustainable Territorial Development," Sustainability, MDPI, vol. 13(21), pages 1-32, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:59:y:2025:i:1:d:10.1007_s11135-024-01950-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.