Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters

Introduction: Large language models are increasingly evaluated on medical examination questions, but their performance on dental questions, and the robustness of that performance to question format and decoding settings, is not established. Methods: A fixed set of 100 dental multiple-choice questions was drawn from the MedMCQA validation split and posed to a large language model in a zero-shot setting. Accuracy was scored against the reference answer, and sensitivity analyses varied question format, decoding temperature and sample size. Results: The model answered 70 of 100 dental examination questions correctly, for an accuracy of 0.700 (95% CI 0.604 to 0.781). The estimate remained essentially unchanged at a higher decoding temperature (0.690) and on a 50-question subsample (0.700), while it differed between single-choice (0.733) and multiple-choice (0.625) question formats. Conclusions: The model achieved a robust accuracy of 0.700 on dental examination questions, and the estimate was stable to temperature and sample size but sensitive to question format, with higher accuracy on single-choice than on multiple-choice questions.