Reinforced Correlation Between Vision and Language for Precise Medical AI Assistant

TL;DR

RCMed tackles the core challenge of reliable vision–language alignment in medical AI by enforcing a self‑reinforcing loop where visual features inform language context and vice versa, augmented by Color Region Description (CRD) to richly describe anatomy. Trained on a massive 20M image–mask–description dataset (RCMedData) across 9 modalities and 177 tasks, RCMed achieves state‑of‑the‑art segmentation precision and strong generalization to external datasets and diverse populations. The model functions as a full‑stack assistant for detection, diagnosis, and segmentation, delivering rapid, text‑driven outputs with built‑in interpretability and audit trails. These capabilities advance human‑centric AI in healthcare by reducing expert dependency, enabling broader clinical adoption, and providing robust, explainable multimodal reasoning across complex medical scenarios.

Abstract

Medical AI assistants support doctors in disease diagnosis, medical image analysis, and report generation. However, they still face significant challenges in clinical use, including limited accuracy with multimodal content and insufficient validation in real-world settings. We propose RCMed, a full-stack AI assistant that improves multimodal alignment in both input and output, enabling precise anatomical delineation, accurate localization, and reliable diagnosis through hierarchical vision-language grounding. A self-reinforcing correlation mechanism allows visual features to inform language context, while language semantics guide pixel-wise attention, forming a closed loop that refines both modalities. This correlation is enhanced by a color region description strategy, translating anatomical structures into semantically rich text to learn shape-location-text relationships across scales. Trained on 20 million image-mask-description triplets, RCMed achieves state-of-the-art precision in contextualizing irregular lesions and subtle anatomical boundaries, excelling in 165 clinical tasks across 9 modalities. It achieved a 23.5% relative improvement in cell segmentation from microscopy images over prior methods. RCMed's strong vision-language alignment enables exceptional generalization, with state-of-the-art performance in external validation across 20 clinically significant cancer types, including novel tasks. This work demonstrates how integrated multimodal models capture fine-grained patterns, enabling human-level interpretation in complex scenarios and advancing human-centric AI healthcare.

Figures (14)

• Figure 1: a, Our RCMed system performs three key stages of medical image analysis: detection, diagnosis, and segmentation. It takes multi-modal inputs—including clinician-provided text and patient medical images across various modalities—and generates comprehensive multi-modal analysis results. b, Overview of the RCMedData, which consists of 20 million image-mask-description triplets for training and includes a comprehensive external evaluation set featuring severe cancers. c, In medical imaging, a textual label like "liver tumor" may refer to tumors with diverse shapes and locations. Consequently, relying solely on this prompt does not convey the detailed morphological information necessary to align with the varied features present in the images. Our RCMed addresses this challenge by establishing a robust correlation between vision and language. We generate detailed and specific descriptions that effectively capture the morphological information inherent in the images in the training stage. d, Visual comparison samples from the held-out evaluation set.
• Figure 1: Data composition. We collected 20 million public data across nine modalities and 177 segmentation tasks, as illustrated in the outer ring, which is the largest dataset for language-driven segmentation. More importantly, we focus on analyzing severe diseases and cancers, thus collecting 20 disease datasets from hospitals in China and Egypt independently, as illustrated in the inner ring, which can better evaluate the clinical value of the models.
• Figure 2: Performance comparison on held-out evaluation dataset with 835k images. a, Our RCMed requires minimal medical knowledge while demonstrating better segmentation results than other methods. BiomedParse has the same level of medical knowledge but worse performance, and MedSAM requires user prompts for disease regions. Significance levels at which RCMed outperforms the best-competing method, with two-sided paired t-test are **$P<1\times10^{-2}$ and ****$P < 1\times10^{-3}$. Exact $P$ values for the comparison between RCMed and others are: $P < 1.21 \times 10^{-24}$ for MedSAM (no prompt); $P < 3.41 \times 10^{-19}$ for MedSAM (1 point); $P <5.12 \times 10^{-16}$ for BiomedParse; and $P < 1.61 \times 10^{-3}$ for MedSAM (loose box). b, Comparison across different modalities and segmentation categories. c, Classification (right) and localization (left) performance comparison with medical vision language foundation models (LLaVA-Med li2024llavamed and MedRegA LVFM_wang2024medrega for diagnosis, and MedRPG chen2023medrpg for localization).
• Figure 2: Effectiveness of the Color Region Description (CRD) strategy. Examples across nine modalities show that the CRD strategy can produce comprehensive and accurate shape and relative location information. Trained on the generated image-mask-description triplets, RCMed established strong correlations between language and vision, leading to better handling of diverse morphological variants in disease samples of a specific category.
• Figure 3: Segmentation performance comparison on external datasets. a, Overall results, RCMed outperforms BiomedParse by 17.35% in significance level of **$P<10^{-2}$ ($P=6.93\times 10^{-4}$). b, Category-level comparison across 4 public external datasets and 2 in-house datasets. c, Detailed comparison of all the categories, the categories with red color highlighted are the unseen categories in the training process. d&e, Qualitative comparison with BiomedParse and MedSAM (loose box).