Comparative study of the performance of ChatGPT-4, Claude, Gemini, Mistral, and perplexity on multiple-choice questions in cardiology

---

Résumé

Objective
The objective of our study was to assess and compare the performance of five LLMs on multiple-choice questions (MCQs) in cardiology.

Materials and methods
This was a comparative study conducted in the cardiology department of the Bogodogo University Hospital, Ouagadougou, involving 83 MCQs derived from the 2020 French national cardiology curriculum. The questions were submitted to ChatGPT-4, Claude, Gemini, Mistral, and Perplexity. Performance was evaluated based on overall and thematic accuracy, as well as the number of discordances. Agreement between the LLMs was assessed using the Kruskal–Wallis test.

Results
Claude achieved the highest overall accuracy (78.31%), followed by ChatGPT-4 and Gemini (75.90%), then Mistral (72.29%) and Perplexity (68.67%). Each LLM demonstrated a distinct performance profile by topic, with Claude excelling in heart failure (100%) and arrhythmias (90.9%), and ChatGPT-4 in diagnostic investigations (87.5%). The analysis of discordances showed a slightly higher precision for ChatGPT-4. The Kruskal–Wallis test with effect size revealed statistically significant differences in performance between the LLMs, whether globally, by topic (p < 0.05) and with generally large effect sizes.

Conclusion
Despite variations in their performance profiles, these five LLMs studied have relatively similar capabilities for answering well-structured cardiology multiple-choice questions. They could therefore be valuable tools in medical education in our resource-limited context.

Mots-clés

Artificial intelligence, Large language model, Cardiology, Medical education, Multiple-choice question, Ouagadougou, Burkina faso