Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration

Yakun Zhu, Yutong Huang, Shengqian Qin, Zhongzhen Huang, Shaoting Zhang, Xiaofan Zhang

TL;DR

MedMCP-Calc tackles realism gaps in medical calculator benchmarks by introducing an MCP-enabled framework that enables realistic, multi-step clinical workflows with active EHR data access and external tool use. It formalizes the problem as a $POMDP$ and defines four core components and comprehensive evaluation metrics to assess end-to-end task fulfillment, calculator selection, numerical precision, and evidence acquisition. A broad evaluation across 23 models reveals significant challenges in planning, tool use, and data retrieval, with domain-dependent performance. CalcMate demonstrates that scenario planning combined with aggressive tool augmentation can achieve state-of-the-art results among open-source models and close the gap with proprietary systems, underscoring the value of targeted training for clinical workflow reasoning.

Abstract

Medical calculators are fundamental to quantitative, evidence-based clinical practice. However, their real-world use is an adaptive, multi-stage process, requiring proactive EHR data acquisition, scenario-dependent calculator selection, and multi-step computation, whereas current benchmarks focus only on static single-step calculations with explicit instructions. To address these limitations, we introduce MedMCP-Calc, the first benchmark for evaluating LLMs in realistic medical calculator scenarios through Model Context Protocol (MCP) integration. MedMCP-Calc comprises 118 scenario tasks across 4 clinical domains, featuring fuzzy task descriptions mimicking natural queries, structured EHR database interaction, external reference retrieval, and process-level evaluation. Our evaluation of 23 leading models reveals critical limitations: even top performers like Claude Opus 4.5 exhibit substantial gaps, including difficulty selecting appropriate calculators for end-to-end workflows given fuzzy queries, poor performance in iterative SQL-based database interactions, and marked reluctance to leverage external tools for numerical computation. Performance also varies considerably across clinical domains. Building on these findings, we develop CalcMate, a fine-tuned model incorporating scenario planning and tool augmentation, achieving state-of-the-art performance among open-source models. Benchmark and Codes are available in https://github.com/SPIRAL-MED/MedMCP-Calc.

MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration

TL;DR

MedMCP-Calc tackles realism gaps in medical calculator benchmarks by introducing an MCP-enabled framework that enables realistic, multi-step clinical workflows with active EHR data access and external tool use. It formalizes the problem as a and defines four core components and comprehensive evaluation metrics to assess end-to-end task fulfillment, calculator selection, numerical precision, and evidence acquisition. A broad evaluation across 23 models reveals significant challenges in planning, tool use, and data retrieval, with domain-dependent performance. CalcMate demonstrates that scenario planning combined with aggressive tool augmentation can achieve state-of-the-art results among open-source models and close the gap with proprietary systems, underscoring the value of targeted training for clinical workflow reasoning.

Abstract

Medical calculators are fundamental to quantitative, evidence-based clinical practice. However, their real-world use is an adaptive, multi-stage process, requiring proactive EHR data acquisition, scenario-dependent calculator selection, and multi-step computation, whereas current benchmarks focus only on static single-step calculations with explicit instructions. To address these limitations, we introduce MedMCP-Calc, the first benchmark for evaluating LLMs in realistic medical calculator scenarios through Model Context Protocol (MCP) integration. MedMCP-Calc comprises 118 scenario tasks across 4 clinical domains, featuring fuzzy task descriptions mimicking natural queries, structured EHR database interaction, external reference retrieval, and process-level evaluation. Our evaluation of 23 leading models reveals critical limitations: even top performers like Claude Opus 4.5 exhibit substantial gaps, including difficulty selecting appropriate calculators for end-to-end workflows given fuzzy queries, poor performance in iterative SQL-based database interactions, and marked reluctance to leverage external tools for numerical computation. Performance also varies considerably across clinical domains. Building on these findings, we develop CalcMate, a fine-tuned model incorporating scenario planning and tool augmentation, achieving state-of-the-art performance among open-source models. Benchmark and Codes are available in https://github.com/SPIRAL-MED/MedMCP-Calc.
Paper Structure (24 sections, 4 equations, 3 figures, 13 tables)

This paper contains 24 sections, 4 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. MedMCP-Calc Benchmark
  4. Formalization
  5. Component
  6. Pipeline
  7. Characteristics
  8. Experiments
  9. Settings
  10. Main Results
  11. CalcMate
  12. Tool Utilization
  13. Conclusion
  14. Experiments Details
  15. Models
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Existing benchmarks typically provide direct instructions with static case data for single-step computation. In contrast, MedMCP-Calc presents fuzzy task descriptions mimicking natural clinician queries, requires multi-step decision making across complex scenarios, enables dynamic interaction with structured EHR databases, and integrates MCP-based tools for physician assistants.
  • Figure 2: Overview of the MedMCP-Calc benchmark. (a) The data construction pipeline comprises four stages: Scenario Instantiation, Task Creation, Database Construction, and Quality Verification. (b) Task execution is formulated as a POMDP, where an LLM agent interacts with MCP servers to process clinical calculator tasks over realistic EHR databases.
  • Figure 3: Extra Calculators Found.