Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

Xiyao Wang, Xiaoyu Tan, Yang Dai, Yuxuan Fu, Shuo Li, Xihe Qiu

TL;DR

VIVID-Med is introduced, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs) via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization.

Abstract

Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.

VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs

TL;DR

VIVID-Med is introduced, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs) via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization.

Abstract

Vision-language pretraining has driven significant progress in medical image analysis. However, current methods typically supervise visual encoders using one-hot labels or free-form text, neither of which effectively captures the complex semantic relationships among clinical findings. In this study, we introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs). VIVID-Med translates clinical findings into verifiable JSON field-state pairs via a Unified Medical Schema (UMS), utilizing answerability-aware masking to focus optimization. It then employs Structured Prediction Decomposition (SPD) to partition cross-attention into orthogonality-regularized query groups, extracting complementary visual aspects. Crucially, the LLM is discarded post-training, yielding a lightweight, deployable ViT-only backbone. We evaluated VIVID-Med across multiple settings: on CheXpert linear probing, it achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data. It also demonstrates robust zero-shot cross-domain transfer to NIH ChestX-ray14 (0.7225 macro-AUC) and strong cross-modality generalization to CT, achieving 0.8413 AUC on LIDC-IDRI lung nodule classification and 0.9969 macro-AUC on OrganAMNIST 11-organ classification. VIVID-Med offers a highly efficient, scalable alternative to deploying resource-heavy vision-language models in clinical settings.
Paper Structure (10 sections, 5 equations, 4 figures, 3 tables)

This paper contains 10 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Unified Medical Schema (UMS) Supervision
  4. Structured Prediction Decomposition (SPD)
  5. Training Objective and Inference
  6. Experiments and Results
  7. CXR Representation and Cross-Domain Transfer
  8. Cross-Modality Transfer to CT
  9. Ablation Studies
  10. Discussion and Conclusion

Figures (4)

  • Figure 1: VIVID-Med overview. The frozen LLM provides structured token supervision during training only. Gradients update the parameters of the ViT and SPD projector, while the deployed model retains only the lightweight ViT backbone.
  • Figure 2: Structured Prediction Decomposition (SPD). Multiple query groups perform cross-attention over shared ViT tokens. Orthogonality regularization encourages complementary branches before shared projection to the LLM embedding space.
  • Figure 3: SPD cross-attention maps for a frontal (top) and lateral (bottom) CXR. Each group attends to distinct anatomical regions.
  • Figure 4: t-SNE of CLS embeddings on CheXpert. VIVID-Med produces tighter, more separable clusters than ImageNet supervised and BiomedCLIP baselines.