Enhancing Medical Large Vision-Language Models via Alignment Distillation

TL;DR

This work tackles hallucinations in medical vision-language models by diagnosing two core issues: insufficient visual representation learning and misaligned visual attention. It introduces MedAlign, a lightweight alignment-distillation framework that transfers domain-specific visual representations and attention patterns from expert CLIP models to Med-LVLMs using two losses, implemented via LoRA at a chosen layer. The approach yields consistent improvements in medical report generation and VQA while enhancing visual grounding and interpretability, demonstrated across multiple datasets and expert CLIPs. The results establish a practical path to more reliable Med-LVLMs without full encoder retraining or extensive supervision.

Abstract

Medical Large Vision-Language Models (Med-LVLMs) have shown promising results in clinical applications, but often suffer from hallucinated outputs due to misaligned visual understanding. In this work, we identify two fundamental limitations contributing to this issue: insufficient visual representation learning and poor visual attention alignment. To address these problems, we propose MEDALIGN, a simple, lightweight alignment distillation framework that transfers visual alignment knowledge from a domain-specific Contrastive Language-Image Pre-training (CLIP) model to Med-LVLMs. MEDALIGN introduces two distillation losses: a spatial-aware visual alignment loss based on visual token-level similarity structures, and an attention-aware distillation loss that guides attention toward diagnostically relevant regions. Extensive experiments on medical report generation and medical visual question answering (VQA) benchmarks show that MEDALIGN consistently improves both performance and interpretability, yielding more visually grounded outputs.

Figures (11)

• Figure 1: A $t$-SNE visualization of visual features derived from sampled abdominal CT scans using LLaVA-Med-1.5 and UniMed-CLIP.
• Figure 2: Attention visualizations from LLaVA-Med-1.5 and UniMed-CLIP on an abdominal CT scan. The red box (not part of the input) highlights the liver cancer region.
• Figure 3: The overview of the proposed MedAlign that uses an expert CLIP model as a reference to guide the fine-tuning of Med-LVLMs via a spatial-aware visual alignment loss and an attention-aware distillation loss.
• Figure 4: Ablation study on distillation design. (A) Impact of removing loss designs. (B) Performance when directly using UniMed-CLIP features without distillation ("w/o Distill").
• Figure 5: A $t$-SNE visualization of visual features from multiple layers of LLaVA-Med-1.5 after applying MedAlign.