Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models

Shaonan Liu, Guo Yu, Xiaoling Luo, Shiyi Zheng, Wenting Chen, Jie Liu, Linlin Shen

TL;DR

MedGaze-Bench introduces the first benchmark that uses clinician gaze as a Cognitive Cursor to evaluate egocentric clinical intent understanding in Medical Multimodal LLMs across open surgery, breech delivery simulation, and diagnostic radiology. The authors propose a Three-Dimensional Clinical Intent Framework (Spatial, Temporal, Standard) and a Trap QA protocol to stress-test perceptual and cognitive reliability, complemented by gaze-anchored prompting to ground model reasoning in visual evidence. Evaluations across nine MLLMs reveal that larger models fare better on basic competency but still struggle with backward causal reasoning and safety-critical decision-making, with reliability under the Trap QA threshold and mixed benefits from gaze prompting. The work highlights critical gaps between current Med-MLLM capabilities and real-world clinical reliability, underscoring the need for architectural innovations that integrate precise visual grounding, causal inference, and robust safety logic for deployment in clinical settings.

Abstract

Medical Multimodal Large Language Models (Med-MLLMs) require egocentric clinical intent understanding for real-world deployment, yet existing benchmarks fail to evaluate this critical capability. To address these challenges, we introduce MedGaze-Bench, the first benchmark leveraging clinician gaze as a Cognitive Cursor to assess intent understanding across surgery, emergency simulation, and diagnostic interpretation. Our benchmark addresses three fundamental challenges: visual homogeneity of anatomical structures, strict temporal-causal dependencies in clinical workflows, and implicit adherence to safety protocols. We propose a Three-Dimensional Clinical Intent Framework evaluating: (1) Spatial Intent: discriminating precise targets amid visual noise, (2) Temporal Intent: inferring causal rationale through retrospective and prospective reasoning, and (3) Standard Intent: verifying protocol compliance through safety checks. Beyond accuracy metrics, we introduce Trap QA mechanisms to stress-test clinical reliability by penalizing hallucinations and cognitive sycophancy. Experiments reveal current MLLMs struggle with egocentric intent due to over-reliance on global features, leading to fabricated observations and uncritical acceptance of invalid instructions.

Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models

TL;DR

MedGaze-Bench introduces the first benchmark that uses clinician gaze as a Cognitive Cursor to evaluate egocentric clinical intent understanding in Medical Multimodal LLMs across open surgery, breech delivery simulation, and diagnostic radiology. The authors propose a Three-Dimensional Clinical Intent Framework (Spatial, Temporal, Standard) and a Trap QA protocol to stress-test perceptual and cognitive reliability, complemented by gaze-anchored prompting to ground model reasoning in visual evidence. Evaluations across nine MLLMs reveal that larger models fare better on basic competency but still struggle with backward causal reasoning and safety-critical decision-making, with reliability under the Trap QA threshold and mixed benefits from gaze prompting. The work highlights critical gaps between current Med-MLLM capabilities and real-world clinical reliability, underscoring the need for architectural innovations that integrate precise visual grounding, causal inference, and robust safety logic for deployment in clinical settings.

Abstract

Medical Multimodal Large Language Models (Med-MLLMs) require egocentric clinical intent understanding for real-world deployment, yet existing benchmarks fail to evaluate this critical capability. To address these challenges, we introduce MedGaze-Bench, the first benchmark leveraging clinician gaze as a Cognitive Cursor to assess intent understanding across surgery, emergency simulation, and diagnostic interpretation. Our benchmark addresses three fundamental challenges: visual homogeneity of anatomical structures, strict temporal-causal dependencies in clinical workflows, and implicit adherence to safety protocols. We propose a Three-Dimensional Clinical Intent Framework evaluating: (1) Spatial Intent: discriminating precise targets amid visual noise, (2) Temporal Intent: inferring causal rationale through retrospective and prospective reasoning, and (3) Standard Intent: verifying protocol compliance through safety checks. Beyond accuracy metrics, we introduce Trap QA mechanisms to stress-test clinical reliability by penalizing hallucinations and cognitive sycophancy. Experiments reveal current MLLMs struggle with egocentric intent due to over-reliance on global features, leading to fabricated observations and uncritical acceptance of invalid instructions.
Paper Structure (22 sections, 4 figures, 5 tables)

This paper contains 22 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General VideoQA Benchmarks
  4. Medical VideoQA Benchmarks
  5. Gaze in Clinical Cognition
  6. MedGaze-Bench
  7. Design Philosophy: Three-Dimensional Intent Framework
  8. Data Collection and Curation
  9. Evaluation Strategy
  10. Experiments
  11. Experimental setup
  12. Experimental Results
  13. Discussion
  14. Conclusion
  15. Limitations
  16. ...and 7 more sections

Figures (4)

  • Figure 1: MedGaze-Bench with three-dimensional clinical intent understanding capability evaluation from spatial, temporal to medical standard.
  • Figure 2: MedGaze-Bench, a three-dimensional evaluation system for clinical intent understanding based on gaze tracking across four medical scenarios. It assesses spatial intent (Where?), temporal intent (Why?), and standard intent understanding (How?), using gaze as a cognitive proxy to evaluate MLLMs' ability to bridge visual perception and clinical decision-making.
  • Figure 3: (a) MedGaze-Bench categories. (b) Data distribution across 4 clinical scenarios.
  • Figure 4: Qualitative evaluation on three key clinical intent understanding tasks with six fine-grained sub-tasks.