Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ART: Action-based Reasoning Task Benchmarking for Medical AI Agents

Ananya Mantravadi, Shivali Dalmia, Abhishek Mukherji

TL;DR

ART presents an action-based reasoning benchmark for medical AI by mining real EHR data to create tasks that stress data retrieval, temporal aggregation, and threshold-based decisions. It identifies three core failure modes and introduces a four-stage, human-in-the-loop pipeline to generate 600+ clinically grounded tasks across 11 LOINC lab codes with a FHIR interface. In initial evaluations, retrieval is nearly perfect after prompt refinement, yet aggregation and threshold reasoning show notable gaps (roughly 28–64% and 32–38% success, respectively) for state-of-the-art models. The work emphasizes concrete failure modes to guide development of safer, more reliable clinical AI agents and suggests a path toward continuous, QA-driven improvement in healthcare decision support systems.

Abstract

Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over structured electronic health records (EHRs). While large language models (LLMs) show promise in healthcare, existing benchmarks inadequately assess performance on action-based tasks involving threshold evaluation, temporal aggregation, and conditional logic. We introduce ART, an Action-based Reasoning clinical Task benchmark for medical AI agents, which mines real-world EHR data to create challenging tasks targeting known reasoning weaknesses. Through analysis of existing benchmarks, we identify three dominant error categories: retrieval failures, aggregation errors, and conditional logic misjudgments. Our four-stage pipeline -- scenario identification, task generation, quality audit, and evaluation -- produces diverse, clinically validated tasks grounded in real patient data. Evaluating GPT-4o-mini and Claude 3.5 Sonnet on 600 tasks shows near-perfect retrieval after prompt refinement, but substantial gaps in aggregation (28--64%) and threshold reasoning (32--38%). By exposing failure modes in action-oriented EHR reasoning, ART advances toward more reliable clinical agents, an essential step for AI systems that reduce cognitive load and administrative burden, supporting workforce capacity in high-demand care settings

ART: Action-based Reasoning Task Benchmarking for Medical AI Agents

TL;DR

ART presents an action-based reasoning benchmark for medical AI by mining real EHR data to create tasks that stress data retrieval, temporal aggregation, and threshold-based decisions. It identifies three core failure modes and introduces a four-stage, human-in-the-loop pipeline to generate 600+ clinically grounded tasks across 11 LOINC lab codes with a FHIR interface. In initial evaluations, retrieval is nearly perfect after prompt refinement, yet aggregation and threshold reasoning show notable gaps (roughly 28–64% and 32–38% success, respectively) for state-of-the-art models. The work emphasizes concrete failure modes to guide development of safer, more reliable clinical AI agents and suggests a path toward continuous, QA-driven improvement in healthcare decision support systems.

Abstract

Reliable clinical decision support requires medical AI agents capable of safe, multi-step reasoning over structured electronic health records (EHRs). While large language models (LLMs) show promise in healthcare, existing benchmarks inadequately assess performance on action-based tasks involving threshold evaluation, temporal aggregation, and conditional logic. We introduce ART, an Action-based Reasoning clinical Task benchmark for medical AI agents, which mines real-world EHR data to create challenging tasks targeting known reasoning weaknesses. Through analysis of existing benchmarks, we identify three dominant error categories: retrieval failures, aggregation errors, and conditional logic misjudgments. Our four-stage pipeline -- scenario identification, task generation, quality audit, and evaluation -- produces diverse, clinically validated tasks grounded in real patient data. Evaluating GPT-4o-mini and Claude 3.5 Sonnet on 600 tasks shows near-perfect retrieval after prompt refinement, but substantial gaps in aggregation (28--64%) and threshold reasoning (32--38%). By exposing failure modes in action-oriented EHR reasoning, ART advances toward more reliable clinical agents, an essential step for AI systems that reduce cognitive load and administrative burden, supporting workforce capacity in high-demand care settings
Paper Structure (21 sections, 4 figures)

This paper contains 21 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Limitations of Current Evaluation Approaches
  3. Our Contributions
  4. Failure Mode analysis
  5. Data Retrieval Failures
  6. (a) Missing or Incomplete Retrieval.
  7. (b) Incorrect Value Retrieval.
  8. Aggregation and Calculation Errors
  9. Multi-step and Conditional Logic Failures
  10. Proposed Approach
  11. Stage 1 - Scenario Identification Agent:
  12. Data Synthesis:
  13. Retrieval Tasks:
  14. Aggregation Tasks:
  15. Multi-step Conditional Tasks:
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Agentic Task Generation & Evaluation Pipeline
  • Figure 2: Success Rate (SR, %) with 200 tasks per category.
  • Figure 3: Task example for threshold-based failure mode
  • Figure 4: Task Generator Agent response example