IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0312771.html
   My bibliography  Save this article

Performance of ChatGPT in medical licensing examinations in countries worldwide: A systematic review and meta-analysis protocol

Author

Listed:
  • Mingxin Liu
  • Tsuyoshi Okuhara
  • Xinyi Chang
  • Hiroko Okada
  • Takahiro Kiuchi

Abstract

Introduction: In November 2022, the online artificial intelligence (AI) chatbot ChatGPT was released to the public, and swiftly garnered global attention because of its ability to provide detailed answers to complex queries. In medical field, ChatGPT has shown great potential to be used in medical education and has excelled in many English-language medical licensing examinations. However, due to the variability of medical licensing examinations in different countries, and ChatGPT’s particular proficiency in English, the previous literatures showed that ChatGPT is unable to pass medical licensing examinations from non-English-speaking countries or those not administered in English. To the best of our knowledge, this is the first study to review whether ChatGPT can demonstrate consistent accuracy across diverse medical licensing examinations and be used in medical education across countries. Objective: In this study protocol, we aimed to analyze and review the differences in performance of ChatGPT in medical exams in various language environments and countries, as well as its potential in medical education. Methods and analysis: A systematic review and meta-analysis was conducted using PubMed, Web of Science, and Scopus to collect papers testing the performance of ChatGPT in medical licensing examinations. We imported all the collected literatures into Rayyan and screened the literatures based on the selection criteria and exclusion criteria. The risk of bias and quality of included studies was assessed by using Mixed Methods Appraisal Tool (MMAT). Data from included studies was extracted into an Excel spreadsheet. All of the above processes were completed by two reviewers independently. A third reviewer was consulted in cases of disagreement. Finally, we provided both quantitative and qualitative analysis of the findings from the included studies. Trial registration: PROSPERO registration number: CRD42024506687.

Suggested Citation

  • Mingxin Liu & Tsuyoshi Okuhara & Xinyi Chang & Hiroko Okada & Takahiro Kiuchi, 2024. "Performance of ChatGPT in medical licensing examinations in countries worldwide: A systematic review and meta-analysis protocol," PLOS ONE, Public Library of Science, vol. 19(10), pages 1-9, October.
    • Handle: RePEc:plo:pone00:0312771
    DOI: 10.1371/journal.pone.0312771
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0312771
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0312771&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0312771?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0312771. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.