MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents

TL;DR

MedBench v4 tackles the gap between static medical AI benchmarks and real-world clinical deployment by providing a scalable, cloud-based evaluation platform with over 700,000 expert-validated tasks across 24 primary and 91 secondary specialties. It supports three tracks—large language models, multimodal models, and governance-aware agents—through rotating test pools, a multi-stage expert validation pipeline, and an LLM-as-judge calibrated to physician ratings, complemented by substantial human evaluation. The study shows that agent-based orchestration markedly enhances end-to-end clinical readiness, safety, and task performance, while multimodal models exhibit solid perception but weaker cross-modal reasoning, and base LLMs struggle on safety. By aligning tasks with Chinese clinical guidelines and regulatory priorities, MedBench offers a practical reference for hospitals, developers, and policymakers to audit and advance safe, effective AI-enabled healthcare.

Abstract

Recent advances in medical large language models (LLMs), multimodal models, and agents demand evaluation frameworks that reflect real clinical workflows and safety constraints. We present MedBench v4, a nationwide, cloud-based benchmarking infrastructure comprising over 700,000 expert-curated tasks spanning 24 primary and 91 secondary specialties, with dedicated tracks for LLMs, multimodal models, and agents. Items undergo multi-stage refinement and multi-round review by clinicians from more than 500 institutions, and open-ended responses are scored by an LLM-as-a-judge calibrated to human ratings. We evaluate 15 frontier models. Base LLMs reach a mean overall score of 54.1/100 (best: Claude Sonnet 4.5, 62.5/100), but safety and ethics remain low (18.4/100). Multimodal models perform worse overall (mean 47.5/100; best: GPT-5, 54.9/100), with solid perception yet weaker cross-modal reasoning. Agents built on the same backbones substantially improve end-to-end performance (mean 79.8/100), with Claude Sonnet 4.5-based agents achieving up to 85.3/100 overall and 88.9/100 on safety tasks. MedBench v4 thus reveals persisting gaps in multimodal reasoning and safety for base models, while showing that governance-aware agentic orchestration can markedly enhance benchmarked clinical readiness without sacrificing capability. By aligning tasks with Chinese clinical guidelines and regulatory priorities, the platform offers a practical reference for hospitals, developers, and policymakers auditing medical AI.

Figures (3)

• Figure 1: Geographic distribution of clinical partners contributing to MedBench. The map and donut chart show the regional distribution of more than 500 member institutions across China that participated in benchmark item refinement and multi-round clinical auditing, including hospitals, medical societies, and academic centers. East China contributes the largest share of partners, followed by South, Central, North, Southwest, Northwest, and Northeast China.
• Figure 2: Benchmark performance of frontier models on MedBench datasets. Bars show task-specific scores (rescaled to 0--100; higher is better). Models compared include O4-mini, Gemini 2.5 Pro, GPT-4o, Llama 4 Maverick, MedGemma 27B-IT, Grok 4, GPT-5, Claude Sonnet 4.5, Qwen2.5-VL-72B-Instruct, and HuatuoGPT-Vision 34B.
• Figure 3: Overview of MedBench tasks for LLMs, multimodal models, and agents. MedBench groups datasets into three capability layers. LLM tasks cover medical language understanding and generation, clinical reasoning, and safety--ethics--compliance evaluation. Multimodal tasks assess visual perception and cross-modal reasoning across images, documents, and mixed inputs. Agent tasks target tool-augmented and multi-agent systems, evaluating intention recognition, task planning, tool use, long-horizon interaction, and safety-aware behavior in realistic clinical workflows.