Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation

Chengxi Zeng, Yuxuan Jiang, Fan Zhang, Alberto Gambaruto, Tilo Burghardt

TL;DR

This paper tackles the challenge of deploying large medical foundation models for segmentation by introducing a multi-model agglomeration framework that distills knowledge from multiple experts (e.g., MedCLIP, RAD-DINO, MedSAM2) into lightweight encoders. The approach integrates an encoder-decoder architecture and two loss-balancing strategies (MLP-based and attention-based) to fuse diverse teacher representations without adding parameters, achieving an average Dice improvement of approximately $2\%$ over simple distillation on VFSS-5K and enabling up to $187\times$ smaller encoders with competitive segmentation quality. Key findings include the effectiveness of attention-based loss balancing, limited additional gains from incorporating MedCLIP in some ablations, and strong generalization across 12 segmentation tasks. The work offers practical impact by enabling faster, accurate VFSS segmentation with resource-efficient models, with potential applicability to other dense medical imaging tasks.

Abstract

The deployment of foundation models for medical imaging has demonstrated considerable success. However, their training overheads associated with downstream tasks remain substantial due to the size of the image encoders employed, and the inference complexity is also significantly high. Although lightweight variants have been obtained for these foundation models, their performance is constrained by their limited model capacity and suboptimal training strategies. In order to achieve an improved tradeoff between complexity and performance, we propose a new framework to improve the performance of low complexity models via knowledge distillation from multiple large medical foundation models (e.g., MedSAM, RAD-DINO, MedCLIP), each specializing in different vision tasks, with the goal to effectively bridge the performance gap for medical image segmentation tasks. The agglomerated model demonstrates superior generalization across 12 segmentation tasks, whereas specialized models require explicit training for each task. Our approach achieved an average performance gain of 2\% in Dice coefficient compared to simple distillation.

Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation

TL;DR

This paper tackles the challenge of deploying large medical foundation models for segmentation by introducing a multi-model agglomeration framework that distills knowledge from multiple experts (e.g., MedCLIP, RAD-DINO, MedSAM2) into lightweight encoders. The approach integrates an encoder-decoder architecture and two loss-balancing strategies (MLP-based and attention-based) to fuse diverse teacher representations without adding parameters, achieving an average Dice improvement of approximately over simple distillation on VFSS-5K and enabling up to smaller encoders with competitive segmentation quality. Key findings include the effectiveness of attention-based loss balancing, limited additional gains from incorporating MedCLIP in some ablations, and strong generalization across 12 segmentation tasks. The work offers practical impact by enabling faster, accurate VFSS segmentation with resource-efficient models, with potential applicability to other dense medical imaging tasks.

Abstract

The deployment of foundation models for medical imaging has demonstrated considerable success. However, their training overheads associated with downstream tasks remain substantial due to the size of the image encoders employed, and the inference complexity is also significantly high. Although lightweight variants have been obtained for these foundation models, their performance is constrained by their limited model capacity and suboptimal training strategies. In order to achieve an improved tradeoff between complexity and performance, we propose a new framework to improve the performance of low complexity models via knowledge distillation from multiple large medical foundation models (e.g., MedSAM, RAD-DINO, MedCLIP), each specializing in different vision tasks, with the goal to effectively bridge the performance gap for medical image segmentation tasks. The agglomerated model demonstrates superior generalization across 12 segmentation tasks, whereas specialized models require explicit training for each task. Our approach achieved an average performance gain of 2\% in Dice coefficient compared to simple distillation.

Paper Structure

This paper contains 10 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Multi-Model Agglomeration via Distillation
  5. Agglomeration Strategies
  6. Implementation Details
  7. Results and Discussion
  8. Conclusion
  9. Architectural details of the encoders
  10. ACKNOWLEDGEMENTS

Figures (3)

  • Figure 1: Efficient models have shown performance discrepancies compared to fine-tuned foundation model (MedSAM) during distillation; our approach enhances performance without introducing extra parameters. The size of the circles represents the model size.
  • Figure 2: The teacher models' features are agglomerated into student features. The learned representations can be decoded by multiple Light Decoders ($LD$) for general tasks: (a) Clustering visualization, (b) PCA visualization, and (c) SAM-decoded Instance Segmentation.
  • Figure 3: Qualitative results on Pharynx Segmentation comparing Specialist Models to foundation models and our multi-encoder distilled model. The lightweight model evaluated is TinyViT.