VietMed-MCQ: A Consistency-Filtered Data Synthesis Framework for Vietnamese Traditional Medicine Evaluation

TL;DR

VietMed-MCQ tackles the scarcity of high-quality benchmarks for Vietnamese Traditional Medicine by proposing a Retrieval-Augmented Generation framework guarded by a dual-model consistency filter. A Llama-3.1-70B teacher generates context-referenced MCQs from unstructured VTM texts, and a Qwen-2.5-32B student verifies reasoning and grounding to retain $3{,}190$ high-fidelity items, achieving $94.2\%$ expert approval and $\kappa = 0.82$. Benchmarking seven models reveals that cross-lingual transfer from Traditional Chinese Medicine priors can outperform Vietnamese-specialized models, especially at larger scales, though complex diagnostic reasoning remains challenging and MCQ-specific biases persist. The work provides a public dataset and code to accelerate research in low-resource medical NLP, while outlining limitations in the evidence-grounding approach and proposing multimodal and free-response extensions for future improvements.

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in general medical domains. However, their performance significantly degrades in specialized, culturally specific domains such as Vietnamese Traditional Medicine (VTM), primarily due to the scarcity of high-quality, structured benchmarks. In this paper, we introduce VietMed-MCQ, a novel multiple-choice question dataset generated via a Retrieval-Augmented Generation (RAG) pipeline with an automated consistency check mechanism. Unlike previous synthetic datasets, our framework incorporates a dual-model validation approach to ensure reasoning consistency through independent answer verification, though the substring-based evidence checking has known limitations. The complete dataset of 3,190 questions spans three difficulty levels and underwent validation by one medical expert and four students, achieving 94.2 percent approval with substantial inter-rater agreement (Fleiss' kappa = 0.82). We benchmark seven open-source models on VietMed-MCQ. Results reveal that general-purpose models with strong Chinese priors outperform Vietnamese-centric models, highlighting cross-lingual conceptual transfer, while all models still struggle with complex diagnostic reasoning. Our code and dataset are publicly available to foster research in low-resource medical domains.

Figures (4)

• Figure 1: The VietMed-MCQ Data Synthesis Framework. We employ a Teacher model ($\mathcal{M}_T$) to generate candidate questions from medical contexts, followed by a Student model ($\mathcal{M}_S$) that validates the answers blindly. Only samples achieving teacher-student consensus are retained.
• Figure 2: Distribution of issues identified in human validation. Among the 29 flagged questions (5.8% of the 500 validation sample), the most common issues were ambiguous distractor phrasing (38%), outdated medical terminology (29%), and overly technical language requiring clarification (21%). Remaining 12% included factual errors or multiple correct answers.
• Figure 3: Distribution of answer keys. The prevalence of Option B (50.3%) highlights a generation bias common in LLMs. We mitigate this through randomized option shuffling during evaluation to ensure positional invariance.
• Figure 4: Comparative performance of evaluated models across zero-shot and 3-shot settings. The significant gap between Qwen2.5 and others highlights the impact of domain-specific pre-training knowledge over general language adaptation.