Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models

TL;DR

This work tackles the critical challenge of robustness for medical multi-modal LLMs under real-world input perturbations. It introduces RobustMed-Bench to systematically benchmark noise effects across image and text modalities and presents a training-free Inherent-enhanced Multi-modal Calibration (IMC) framework that leverages model-intrinsic denoising capabilities. IMC combines Perturbation-aware Denoising Calibration for visual inputs with a Self-instantiated Multi-agent System for textual denoising, enabling cross-modal robustness without fine-tuning. Extensive experiments on SLAKE- and OmniMed-based data demonstrate state-of-the-art improvements across diverse noise types, advancing the practical clinical applicability of medical MLLMs by reducing vulnerability to artifacts and typos. This work provides a scalable, data-efficient path toward safer deployment of MLLMs in real-world healthcare settings.

Abstract

Medical Multi-modal Large Language Models (MLLMs) have shown promising clinical performance. However, their sensitivity to real-world input perturbations, such as imaging artifacts and textual errors, critically undermines their clinical applicability. Systematic analysis of such noise impact on medical MLLMs remains largely unexplored. Furthermore, while several works have investigated the MLLMs' robustness in general domains, they primarily focus on text modality and rely on costly fine-tuning. They are inadequate to address the complex noise patterns and fulfill the strict safety standards in medicine. To bridge this gap, this work systematically analyzes the impact of various perturbations on medical MLLMs across both visual and textual modalities. Building on our findings, we introduce a training-free Inherent-enhanced Multi-modal Calibration (IMC) framework that leverages MLLMs' inherent denoising capabilities following the perceive-and-calibrate principle for cross-modal robustness enhancement. For the visual modality, we propose a Perturbation-aware Denoising Calibration (PDC) which leverages MLLMs' own vision encoder to identify noise patterns and perform prototype-guided feature calibration. For text denoising, we design a Self-instantiated Multi-agent System (SMS) that exploits the MLLMs' self-assessment capabilities to refine noisy text through a cooperative hierarchy of agents. We construct a benchmark containing 11 types of noise across both image and text modalities on 2 datasets. Experimental results demonstrate our method achieves the state-of-the-art performance across multiple modalities, showing potential to enhance MLLMs' robustness in real clinical scenarios.

Figures (9)

• Figure 1: Analysis of MLLM sensitivity to different input perturbations: (a) MLLM produces incorrect responses when inputs contain noise. (b) Comparison between original and perturbed VQA pairs, which demonstrates the performance deterioration of leading MLLMs under visual and textual perturbations (left: CT sparse view artifact, right: character level typographical errors).
• Figure 2: Overview of SLAKE-based RobustMed-Bench dataset composition: (a) Visualization of noise types included in our benchmark. For images, CT contains low dose and sparse view noise, MRI includes human motion, aliasing and banding corruptions, and X-ray with patient movement artifacts. For text, we incorporate four common typographical errors: add/delete/swap/replace characters in words, and additional unrelated sentences noise; (b) Statistical distribution of modalities and question categories of the SLAKE-based benchmark.
• Figure 3: Architecture of our Inherent-enhanced Multi-modal Calibration (IMC) framework, which follows the perceive-and-calibrate process for both image and text modalities. (a) Perturbation-aware Denoising Calibration (PDC) enables image denoising through prototype-guided classification, then applies PCA-based calibration vectors to rectify vision features across all vision encoder layers; (b) Self-instantiated Multi-agent System (SMS) enables text denoising through hierarchical multi-agent coordination with micro and macro iterations that simulate human iterative editing processes; (c) Our framework integrates PDC and SMS for unified multi-modal denoising.
• Figure 4: Statistical distribution of modalities and question categories of the OmniMed-based benchmark.
• Figure 5: (a-b) Case studies of close-ended and open-ended questions on SLAKE-based CT low dose artifact; (c) Case study of Multiple-Choice Question on OmniMed-based MRI Motion noise.