MultifacetEval: Multifaceted Evaluation to Probe LLMs in Mastering Medical Knowledge

TL;DR

This paper tackles the gap between impressive medical benchmark performance and real-world medical effectiveness of LLMs. It introduces MultifacetEval, a knowledge-centric framework that quantifies medical knowledge mastery across four facets—$\text{Comparison}$, $\text{Rectification}$, $\text{Discrimination}$, and $\text{Verification}$—via the mastery metric $r_k(M)=\prod_{i=1}^N f^i_k(M)$ and overall score $p(M)=\frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}} r_k(M)$. The framework generates multifaceted questions and evaluates 13 LLMs on two datasets, MultiDiseK and MultiMedQA, revealing that real mastery is substantially lower than benchmark performance and varies significantly across facets, with larger models showing greater robustness under multifaceted evaluation. The findings suggest that medical foundation models must be large and trained on diverse medical tasks to achieve true clinical applicability, and the authors release code and datasets to support broader adoption and further research.

Abstract

Large language models (LLMs) have excelled across domains, also delivering notable performance on the medical evaluation benchmarks, such as MedQA. However, there still exists a significant gap between the reported performance and the practical effectiveness in real-world medical scenarios. In this paper, we aim to explore the causes of this gap by employing a multifaceted examination schema to systematically probe the actual mastery of medical knowledge by current LLMs. Specifically, we develop a novel evaluation framework MultifacetEval to examine the degree and coverage of LLMs in encoding and mastering medical knowledge at multiple facets (comparison, rectification, discrimination, and verification) concurrently. Based on the MultifacetEval framework, we construct two multifaceted evaluation datasets: MultiDiseK (by producing questions from a clinical disease knowledge base) and MultiMedQA (by rephrasing each question from a medical benchmark MedQA into multifaceted questions). The experimental results on these multifaceted datasets demonstrate that the extent of current LLMs in mastering medical knowledge is far below their performance on existing medical benchmarks, suggesting that they lack depth, precision, and comprehensiveness in mastering medical knowledge. Consequently, current LLMs are not yet ready for application in real-world medical tasks. The codes and datasets are available at https://github.com/THUMLP/MultifacetEval.

Figures (7)

• Figure 1: GPT-3.5-turbo responding to medical exam problems assessing the same knowledge point but in different formats.
• Figure 2: Principle of the proposed multifaceted evaluation.
• Figure 3: Framework of the proposed multifaceted evaluation approach that evaluates LLMs' medical knowledge mastery from various facets.
• Figure 4: Example of the proposed revision question.
• Figure 5: Aspects of disease-related knowledge in DiseK.