Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation
Shunfan Zheng, Xiechi Zhang, Gerard de Melo, Xiaoling Wang, Linlin Wang
TL;DR
HDCEval tackles misalignment in medical LLM evaluation by proposing a Hierarchical Divide-and-Conquer framework guided by professional medical guidelines. It decomposes complex judgments into specialized subtasks tackled by expert models and trained with Attribute-Driven Token Optimization (ADTO) on a carefully constructed preference dataset, with evaluation outputs expressed as $E=ig\{E_1,...,E_m\big\}$ where each $E_i=(s_i,p_i)$. Across a multisource medical dataset, HDCEval significantly outperforms baselines and shows higher consistency with human evaluators (notably a 23.92% gain over PandaLM), while maintaining robustness to input form variations and reducing model bias via its clever preference-data strategy. The approach yields finer-grained, rationale-supported assessments that better reflect clinical reasoning, suggesting practical impact for safer, more reliable medical AI evaluation in freestyle clinical contexts.
Abstract
In the rapidly evolving landscape of large language models (LLMs) for medical applications, ensuring the reliability and accuracy of these models in clinical settings is paramount. Existing benchmarks often focus on fixed-format tasks like multiple-choice QA, which fail to capture the complexity of real-world clinical diagnostics. Moreover, traditional evaluation metrics and LLM-based evaluators struggle with misalignment, often providing oversimplified assessments that do not adequately reflect human judgment. To address these challenges, we introduce HDCEval, a Hierarchical Divide-and-Conquer Evaluation framework tailored for fine-grained alignment in medical evaluation. HDCEval is built on a set of fine-grained medical evaluation guidelines developed in collaboration with professional doctors, encompassing Patient Question Relevance, Medical Knowledge Correctness, and Expression. The framework decomposes complex evaluation tasks into specialized subtasks, each evaluated by expert models trained through Attribute-Driven Token Optimization (ADTO) on a meticulously curated preference dataset. This hierarchical approach ensures that each aspect of the evaluation is handled with expert precision, leading to a significant improvement in alignment with human evaluators.