Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources

Phuc Pham, Nhu Pham, Ngoc Quoc Ly

TL;DR

The paper tackles the practical constraint of limited annotations and computing resources in medical vision-language pretraining. It introduces momentum self-distillation supported by dual momentum queues and a resource-free batch enlargement method to enable efficient, high-quality cross-modal learning on a single GPU. By replacing exact cross-modal labels with soft similarity targets and leveraging gradient-free momentum keys, the approach achieves competitive zero-shot results and substantial few-shot gains, with superior image-to-text retrieval on medical datasets. This work offers a scalable, hardware-friendly strategy for robust medical VLM development and retrieval tasks, with clear implications for real-world deployment under resource constraints.

Abstract

In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .

Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources

TL;DR

The paper tackles the practical constraint of limited annotations and computing resources in medical vision-language pretraining. It introduces momentum self-distillation supported by dual momentum queues and a resource-free batch enlargement method to enable efficient, high-quality cross-modal learning on a single GPU. By replacing exact cross-modal labels with soft similarity targets and leveraging gradient-free momentum keys, the approach achieves competitive zero-shot results and substantial few-shot gains, with superior image-to-text retrieval on medical datasets. This work offers a scalable, hardware-friendly strategy for robust medical VLM development and retrieval tasks, with clear implications for real-world deployment under resource constraints.

Abstract

In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .

Paper Structure

This paper contains 22 sections, 5 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Setting
  4. Uni-modal Contrastive Learning
  5. Multi-modal Contrastive Learning
  6. Increasing batch size without additional resources
  7. Experiments
  8. Implementation details for evaluations
  9. Augmentation
  10. Configurations
  11. Evaluation details
  12. Datasets
  13. Comparison with state-of-the-arts (SOTA)
  14. Zero-shot Classification
  15. Few-shot Classification
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Our overall framework. Left: Uni-modal learning on images. Right: Multi-modal learning on text-to-image. For simplicity, we illustrate our method using a single sample. The same process applies to uni-modal learning on text and multi-modal learning on image-to-text by substituting the corresponding modules.
  • Figure 2: Our technique to increase batch size without additional resources. We split the primary batch into smaller sub-batches. The preparation of embedding vectors is divided into two separate steps: first, calculating and concatenating the momentum keys from sub-batches, and second, calculating the query vectors and optimizing the contrastive loss with prepared keys. In the second step, Gradient Accumulation is employed to achieve the effects of a large batch size.