MeNTi: Bridging Medical Calculator and LLM Agent with Nested Tool Calling

TL;DR

MeNTi targets the challenge of deploying LLMs for medical calculator tasks by introducing a universal agent that combines a specialized medical toolkit with meta-tool based tool selection and nested tool calling. The CalcQA benchmark, built with real patient cases and 281 tools across 44 calculators, enables end-to-end evaluation of calculator-based clinical reasoning. Experimental results show MeNTi delivering substantial improvements over baselines, particularly in unit conversions and end-to-end calculator calculations, with GPT-4o achieving the highest performance. This work demonstrates a practical path for deploying LLMs in demanding medical calculation contexts and points to future directions in broader downstream medical tasks.

Abstract

Integrating tools into Large Language Models (LLMs) has facilitated the widespread application. Despite this, in specialized downstream task contexts, reliance solely on tools is insufficient to fully address the complexities of the real world. This particularly restricts the effective deployment of LLMs in fields such as medicine. In this paper, we focus on the downstream tasks of medical calculators, which use standardized tests to assess an individual's health status. We introduce MeNTi, a universal agent architecture for LLMs. MeNTi integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization. Specifically, it achieves flexible tool selection and nested tool calling to address practical issues faced in intricate medical scenarios, including calculator selection, slot filling, and unit conversion. To assess the capabilities of LLMs for quantitative assessment throughout the clinical process of calculator scenarios, we introduce CalcQA. This benchmark requires LLMs to use medical calculators to perform calculations and assess patient health status. CalcQA is constructed by professional physicians and includes 100 case-calculator pairs, complemented by a toolkit of 281 medical tools. The experimental results demonstrate significant performance improvements with our framework. This research paves new directions for applying LLMs in demanding scenarios of medicine.

Figures (5)

• Figure 1: Instance of calculator application in Medical Scenarios. Physicians use calculators to assist in the quantitative assessment of patient's status based on their primary assessment.
• Figure 2: The whole workflow of MeNTi: (a) The physician proposes the next step for a more quantitative analysis. (b) Upon receiving the doctor’s request, MeNTi uses the meta-tool to determine and select the appropriate tool from the toolkit. (c) While filling in the parameters, MeNTi detects a unit mismatch and uses the nested tool calling mechanism to identify the need for additional tools. (d) MeNTi solves problems with more tools. (e) MeNTi consolidates the results from all tools and refills the parameters. (f) The final result is computed.
• Figure 3: Distribution map of the assessment calculator departments. The calculator used in this study encompasses 13 clinical departments, with a particular emphasis on cardiology, involving 11 calculators.
• Figure 4: Flow of Nested Tool Calling. MeNTi enhances LLMs' tool utilization capabilities using meta-tool and nested calling mechanism, while also extending the knowledge boundaries of LLMs with a specialized medical toolkit, complementing each other. Through the nested tool calling of the toolkit, MeNTi provides LLMs with more operation and information for solving practical tasks.
• Figure 5: Distribution map of the assessment calculator departments.