How Well Did ChatGPT Perform in Answering Questions on Different Topics in Gross Anatomy?

The burgeoning interest in leveraging ChatGPT within the medical field underscores the necessity for a comprehensive understanding of its capabilities and limitations, particularly in the context of medical assessments and examinations. The model possesses a unique aptitude for addressing queries related to medical student exams, thereby serving as an invaluable resource for academic support. Its advanced natural language processing capabilities empower it to comprehend the intricacies of medical terminology, enabling it to provide nuanced and contextually relevant responses. This study aimed to quantitatively evaluate ChatGPT performance in answering Multiple Choice Questions (MCQs) related to the different topics in Gross Anatomy course for medical students. The research conducted for this study was focused on a comprehensive examination of ChatGPT (GPT-3.5) capabilities in answering 325 MCQs designed in USMLE style, arranged in 7 different sets related to specific topics. These questions were selected from Gross Anatomy course exam database for medical students and reviewed by three independent experts. The results of 5 successive attempts to answer each set of questions by Chat-GPT were evaluated based on accuracy, relevance, and comprehensiveness. The ChatGPT provided accurate answers to 44.1% ± 8.2% of questions. Accordingly, to our data, ChatGPT is answering much better on MCQs from Back material (58.4%), following Head and Neck (48.8%) and Pelvis (45.6%), and performing not so well in questions of Thorax (37.6%) and Upper limb (36.4%). ChatGPT is struggling in answering questions about blood supply and innervation of the specific organs. ChatGPT stands out as a promising and interactive educational tool, particularly for students engaged in the study of anatomy. Its distinctive ability to not only provide informative responses but also engage students in a conversational manner is highly commendable. This quality has the potential to enhance student engagement and foster curiosity, creating a dynamic learning experience. However, it’s crucial to acknowledge that ChatGPT’s current level of comprehension and interpretative abilities may not meet the demanding standards required for practical applications in the medical education domain. Its performance in challenging examinations like medical college exams and health licensing exams might need to catch up to expectations.