IDEAS home Printed from https://ideas.repec.org/a/spr/pharmo/v8y2024i2d10.1007_s41669-024-00477-8.html
   My bibliography  Save this article

Artificial Intelligence to Automate Health Economic Modelling: A Case Study to Evaluate the Potential Application of Large Language Models

Author

Listed:
  • Tim Reason

    (Estima Scientific)

  • William Rawlinson

    (Estima Scientific)

  • Julia Langham

    (Estima Scientific)

  • Andy Gimblett

    (Estima Scientific)

  • Bill Malcolm

    (Bristol Myers Squibb)

  • Sven Klijn

    (Bristol Myers Squibb)

Abstract

Background Current generation large language models (LLMs) such as Generative Pre-Trained Transformer 4 (GPT-4) have achieved human-level performance on many tasks including the generation of computer code based on textual input. This study aimed to assess whether GPT-4 could be used to automatically programme two published health economic analyses. Methods The two analyses were partitioned survival models evaluating interventions in non-small cell lung cancer (NSCLC) and renal cell carcinoma (RCC). We developed prompts which instructed GPT-4 to programme the NSCLC and RCC models in R, and which provided descriptions of each model’s methods, assumptions and parameter values. The results of the generated scripts were compared to the published values from the original, human-programmed models. The models were replicated 15 times to capture variability in GPT-4’s output. Results GPT-4 fully replicated the NSCLC model with high accuracy: 100% (15/15) of the artificial intelligence (AI)-generated NSCLC models were error-free or contained a single minor error, and 93% (14/15) were completely error-free. GPT-4 closely replicated the RCC model, although human intervention was required to simplify an element of the model design (one of the model’s fifteen input calculations) because it used too many sequential steps to be implemented in a single prompt. With this simplification, 87% (13/15) of the AI-generated RCC models were error-free or contained a single minor error, and 60% (9/15) were completely error-free. Error-free model scripts replicated the published incremental cost-effectiveness ratios to within 1%. Conclusion This study provides a promising indication that GPT-4 can have practical applications in the automation of health economic model construction. Potential benefits include accelerated model development timelines and reduced costs of development. Further research is necessary to explore the generalisability of LLM-based automation across a larger sample of models.

Suggested Citation

  • Tim Reason & William Rawlinson & Julia Langham & Andy Gimblett & Bill Malcolm & Sven Klijn, 2024. "Artificial Intelligence to Automate Health Economic Modelling: A Case Study to Evaluate the Potential Application of Large Language Models," PharmacoEconomics - Open, Springer, vol. 8(2), pages 191-203, March.
    • Handle: RePEc:spr:pharmo:v:8:y:2024:i:2:d:10.1007_s41669-024-00477-8
    DOI: 10.1007/s41669-024-00477-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s41669-024-00477-8
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1007/s41669-024-00477-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Drummond, Michael F. & Sculpher, Mark J. & Torrance, George W. & O'Brien, Bernie J. & Stoddart, Greg L., 2005. "Methods for the Economic Evaluation of Health Care Programmes," OUP Catalogue, Oxford University Press, edition 3, number 9780198529453, Decembrie.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Saha, Sanjib & Gerdtham, Ulf-G. & Toresson, Håkan & Minthon, Lennart & Jarl, Johan, 2018. "Economic Evaluation of Interventions for Screening of Dementia," Working Papers 2018:20, Lund University, Department of Economics.
    2. Mark Oppe & Daniela Ortín-Sulbarán & Carlos Vila Silván & Anabel Estévez-Carrillo & Juan M. Ramos-Goñi, 2021. "Cost-effectiveness of adding Sativex® spray to spasticity care in Belgium: using bootstrapping instead of Monte Carlo simulation for probabilistic sensitivity analyses," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 22(5), pages 711-721, July.
    3. Ties Hoomans & Johan Severens & Nicole Roer & Gepke Delwel, 2012. "Methodological Quality of Economic Evaluations of New Pharmaceuticals in the Netherlands," PharmacoEconomics, Springer, vol. 30(3), pages 219-227, March.
    4. Khan, Mohammed Tajuddin & Kishore, Avinash & Joshi, Pramod K., 2016. "Gender dimensions on farmers’ preferences for direct-seeded rice with drum seeder in India," IFPRI discussion papers 1550, International Food Policy Research Institute (IFPRI).
    5. Daphne C. Voormolen & Judith A. M. Bom & Esther W. Bekker-Grob & Werner B. F. Brouwer & Job Exel, 2024. "Development and Content Validation of the 10-item Well-being Instrument (WiX) for use in Economic Evaluation Studies," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 19(2), pages 381-413, April.
    6. Jose L Burgos & Thomas L Patterson & Joshua S Graff-Zivin & James G Kahn & M Gudelia Rangel & M Remedios Lozada & Hugo Staines & Steffanie A Strathdee, 2016. "Cost-Effectiveness of Combined Sexual and Injection Risk Reduction Interventions among Female Sex Workers Who Inject Drugs in Two Very Distinct Mexican Border Cities," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-15, February.
    7. Najmiatul Fitria & Antoinette D. I. Asselt & Maarten J. Postma, 2019. "Cost-effectiveness of controlling gestational diabetes mellitus: a systematic review," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(3), pages 407-417, April.
    8. Thomas Grochtdreis & Hans-Helmut König & Alexander Dobruschkin & Gunhild von Amsberg & Judith Dams, 2018. "Cost-effectiveness analyses and cost analyses in castration-resistant prostate cancer: A systematic review," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-25, December.
    9. Kim Jeong & John Cairns, 2013. "Review of economic evidence in the prevention and early detection of colorectal cancer," Health Economics Review, Springer, vol. 3(1), pages 1-10, December.
    10. Fleurbaey, Marc & Zuber, Stéphane, 2017. "Fair management of social risk," Journal of Economic Theory, Elsevier, vol. 169(C), pages 666-706.
    11. Boone, Jan, 2015. "Basic versus supplementary health insurance: Moral hazard and adverse selection," Journal of Public Economics, Elsevier, vol. 128(C), pages 50-58.
    12. Eleanor Heather & Katherine Payne & Mark Harrison & Deborah Symmons, 2014. "Including Adverse Drug Events in Economic Evaluations of Anti-Tumour Necrosis Factor-α Drugs for Adult Rheumatoid Arthritis: A Systematic Review of Economic Decision Analytic Models," PharmacoEconomics, Springer, vol. 32(2), pages 109-134, February.
    13. Manuel Gomes & Robert Aldridge & Peter Wylie & James Bell & Owen Epstein, 2013. "Cost-Effectiveness Analysis of 3-D Computerized Tomography Colonography Versus Optical Colonoscopy for Imaging Symptomatic Gastroenterology Patients," Applied Health Economics and Health Policy, Springer, vol. 11(2), pages 107-117, April.
    14. Hareth Al-Janabi & Job van Exel & Werner Brouwer & Joanna Coast, 2016. "A Framework for Including Family Health Spillovers in Economic Evaluation," Medical Decision Making, , vol. 36(2), pages 176-186, February.
    15. Fan Yang & Colin Angus & Ana Duarte & Duncan Gillespie & Mark Sculpher & Simon Walker & Susan Griffin, 2021. "Comparing smoking cessation to screening and brief intervention for alcohol in distributional cost effectiveness analysis to explore the sensitivity of results to socioeconomic inequalities characteri," Working Papers 184cherp, Centre for Health Economics, University of York.
    16. Simon Eckermann & Tim Coelli, 2008. "Including quality attributes in a model of health care efficiency: A net benefit approach," CEPA Working Papers Series WP032008, School of Economics, University of Queensland, Australia.
    17. Saha, Sanjib & Gerdtham, Ulf-G. & Toresson, Håkan & Minthon, Lennart & Jarl, Johan, 2018. "Economic Evaluation of Nonpharmacological Interventions for Dementia Patients and their Caregivers - A Systematic Literature Review," Working Papers 2018:10, Lund University, Department of Economics.
    18. Fenna Arnoldussen & Mark J. Koetse & Sander M. de Bruyn & Onno Kuik, 2022. "What Are People Willing to Pay for Social Sustainability? A Choice Experiment among Dutch Consumers," Sustainability, MDPI, vol. 14(21), pages 1-21, November.
    19. Jesse Elliott & Sasha Katwyk & Bláthnaid McCoy & Tammy Clifford & Beth K. Potter & Becky Skidmore & George A. Wells & Doug Coyle, 2019. "Decision Models for Assessing the Cost Effectiveness of Treatments for Pediatric Drug-Resistant Epilepsy: A Systematic Review of Economic Evaluations," PharmacoEconomics, Springer, vol. 37(10), pages 1261-1276, October.
    20. Denise Howel & Suzanne Moffatt & Catherine Haighton & Andrew Bryant & Frauke Becker & Melanie Steer & Sarah Lawson & Terry Aspray & Eugene M G Milne & Luke Vale & Elaine McColl & Martin White, 2019. "Does domiciliary welfare rights advice improve health-related quality of life in independent-living, socio-economically disadvantaged people aged ≥60 years? Randomised controlled trial, economic and p," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-31, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:pharmo:v:8:y:2024:i:2:d:10.1007_s41669-024-00477-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.