MedBridge: Bridging Foundation Vision-Language Models to Medical Image Diagnosis in Chest X-Ray

TL;DR

MedBridge tackles the domain gap between vision-language foundation models and medical chest X-ray diagnosis by introducing a lightweight, modular adaptation framework. It fuses focal sampling, a Query-Encoder with learnable queries, and a Mixture of Experts to re-purpose frozen VLM backbones for multi-label thoracic disease classification with limited data. Across five chest X-ray datasets and three adaptation scenarios, MedBridge consistently outperforms prior VLM adapters, achieving meaningful gains in AUC and data efficiency while remaining hardware-friendly. The approach is VLM-agnostic and scalable, enabling rapid deployment of diverse backbones for medical imaging tasks with modest compute.

Abstract

Recent vision-language foundation models deliver state-of-the-art results in natural image classification, but falter in medical images due to pronounced domain shifts. Training a medical foundation model also requires substantial resources, including extensive annotated data and high computational capacity. To bridge this gap with minimal overhead, we introduce MedBridge, a lightweight multimodal adaptation framework that flexibly re-purposes arbitrary pre-trained foundation VLMs for medical image diagnosis. MedBridge comprises three novel core components. First, a Focal Sampling module that subsamples and extracts high-resolution local regions to capture subtle pathological features, compensating for the limited input resolution of foundation VLMs. Second, a Query-Encoder model with a small set of learnable queries to align the feature maps of frozen VLMs with medical semantics, without requiring retraining of the backbone layers. Third, a Mixture of Experts mechanism, driven by learnable queries, harnesses the complementary strength of various VLMs to maximize diagnostic performance. We evaluate MedBridge on five chest radiograph benchmarks in three key adaptation tasks, demonstrating its superior performance in both cross-domain and in-domain adaptation settings under varying levels of training data availability. MedBridge achieved an improvement of 6-15% in AUC compared to state-of-the-art VLM adaptation methods in multi-label thoracic disease diagnosis, underscoring its effectiveness in leveraging diverse foundation models for accurate and data-efficient medical diagnosis. Our project and code are available at https://github.com/ai-med/MedBridge.

Figures (4)

• Figure 1: (a) To predict the label(s) of an image $x$, i.e., $p(y|x)$, MedBridge operates in three stages. In the first stage, it expands the visual observations from a single image $x$ to $N$ images $\mathbf{x}$ through a focal sampling module. We then introduce a mixture of experts (MoE), where experts are frozen foundation VLMs. Each expert produces a set of $\mathbf{M}$ tokens for the input image, and simultaneously generates a set of $\mathbf{Q}$ learnable queries that interact with frozen tokens, where $\text{Expert}\;A:\{\mathbf{M}_A, \mathbf{Q}_A\}$ and $\text{Expert}\;B: \{\mathbf{M}_B, \mathbf{Q}_B\}$. A gating network $g$, trained on learnable queries $\mathbf{Q}_G = [\mathbf{Q}_A, \mathbf{Q}_B]$, dynamically weights the experts, allowing prioritization of different components. The final prediction is a combination of the visual tokens $\mathbf{M}$ and the learnable queries $\mathbf{Q}$, modulated by $\alpha$. The learnable queries $\{\mathbf{Q}_A, \mathbf{Q}_B\}$ add approximately 4% more parameters compared to the base models, but they efficiently help guide the MoE toward the target domain's representation. (b) MedBridge yields significant improvements in disease classification across various medical benchmarks.
• Figure 2: MedBridge framework: Focal sampling extracts fine-grained regions from the high-resolution input image, encoded by QEncoders into frozen tokens and learnable queries for lightweight adaptation. A Mixture of Experts (MoE) module routes these tokens through the most relevant encoders, and the final prediction combines soft labels from both query and frozen tokens with weight $\alpha$.
• Figure 3: We evaluate MedBridge in three key adaptation tasks: (1) cross-domain adaptation: adapting foundation VLMs to medical data; (2) in-domain adaptation: adapting pre-trained medical VLMs to new datasets with seen and unseen diseases, and (3) zero-shot cross-dataset adaptation: after being adapted to a medical dataset, we test it on a new dataset without further fine-tuning.
• Figure 4: Ablation on (a) the Focal Sampling module and (b) the number $Q$ and weight $\alpha$ of query tokens in the QEncoder.