MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark
Mouath Abu-Daoud, Leen Kharouf, Omar El Hajj, Dana El Samad, Mariam Al-Omari, Jihad Mallat, Khaled Saleh, Nizar Habash, Farah E. Shamout
TL;DR
MedAraBench introduces a large-scale Arabic medical MCQ benchmark (24,883 items across 19 specialties and five difficulty levels) with train/test splits, validated by expert clinicians and LLM-based judges. The authors benchmark 16 LLMs in zero-shot, and explore few-shot and QLoRA fine-tuning, finding that top proprietary models approach 0.76 accuracy while open-source models struggle, underscoring the need for domain-focused Arabic medical training. The dataset is digitized from regional exams, carefully filtered for quality, and supplemented with a standardized evaluation protocol to enable reproducible cross-model comparisons and drive improvements in multilingual medical NLP. Public release of MedAraBench and evaluation scripts aims to catalyze research on Arabic clinical reasoning and multilingual healthcare AI.
Abstract
Arabic remains one of the most underrepresented languages in natural language processing research, particularly in medical applications, due to the limited availability of open-source data and benchmarks. The lack of resources hinders efforts to evaluate and advance the multilingual capabilities of Large Language Models (LLMs). In this paper, we introduce MedAraBench, a large-scale dataset consisting of Arabic multiple-choice question-answer pairs across various medical specialties. We constructed the dataset by manually digitizing a large repository of academic materials created by medical professionals in the Arabic-speaking region. We then conducted extensive preprocessing and split the dataset into training and test sets to support future research efforts in the area. To assess the quality of the data, we adopted two frameworks, namely expert human evaluation and LLM-as-a-judge. Our dataset is diverse and of high quality, spanning 19 specialties and five difficulty levels. For benchmarking purposes, we assessed the performance of eight state-of-the-art open-source and proprietary models, such as GPT-5, Gemini 2.0 Flash, and Claude 4-Sonnet. Our findings highlight the need for further domain-specific enhancements. We release the dataset and evaluation scripts to broaden the diversity of medical data benchmarks, expand the scope of evaluation suites for LLMs, and enhance the multilingual capabilities of models for deployment in clinical settings.