Prompt-Aware Adaptive Elastic Weight Consolidation for Continual Learning in Medical Vision-Language Models

TL;DR

This work tackles catastrophic forgetting in medical vision-language models by introducing PA-EWC, a continual-learning framework that uses prompt-guided parameter specialization to protect functionally critical parameters. By classifying parameters into Visual-Descriptive, Spatial-Guided, and Medical-Semantic groups and combining adaptive Fisher information with gradient stability and task similarity, PA-EWC achieves targeted knowledge retention while enabling adaptation to new clinical prompts. The approach couples a prompt-aligned complexity metric with a prompt-aware loss to balance current task performance and past task preservation, and it demonstrates superior performance across five diverse medical imaging datasets, including polyp segmentation and chest X-ray localization. The results show reduced forgetting by up to ~17.6 percentage points and competitive Dice gains, highlighting potential for practical deployment in dynamic clinical environments where imaging protocols and terminology evolve.

Abstract

Medical AI systems face catastrophic forgetting when deployed in clinical settings, where models must learn new imaging protocols while retaining prior diagnostic capabilities. This challenge is particularly acute for medical vision-language models that must preserve complex cross-modal alignments between medical images and clinical terminology across diverse imaging modalities. We introduce Prompt- Aware Adaptive Elastic Weight Consolidation (PA-EWC), a novel continual learning approach that addresses catastrophic forgetting through prompt-guided parameter specialization. Our method systematically categorizes model parameters based on their functional roles in processing visual-descriptive, spatial-guided, and medical-semantic information, enabling targeted protection of critical knowledge while allowing adaptation to new clinical requirements. PA-EWC incorporates adaptive Fisher Information computation with gradient stability analysis and develops weighted complexity metrics based on medical terminology density. We evaluate our approach across five medical imaging datasets (Kvasir-SEG, ISIC 2018, CheXlocalize, BUSI, CAMUS) representing diverse modalities including endoscopy, dermoscopy, radiography, and ultrasound. Experimental results demonstrate that PA-EWC reduces catastrophic forgetting by up to 17.58% compared to baseline methods, with performance improvements of 4.30% on chest X-ray pathology localization and 6.06% on polyp segmentation.

Figures (3)

• Figure 1: Overall PA-EWC pipeline
• Figure 2: Comparison of forgetting rates across different prompt strategies. Lower values indicate better performance with reduced catastrophic forgetting.
• Figure 3: Visualizations of segmentation results across different prompt strategies and evaluation metrics.