Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forgetting-Resistant and Lesion-Aware Source-Free Domain Adaptive Fundus Image Analysis with Vision-Language Model

Zheang Huai, Hui Tang, Hualiang Wang, Xiaomeng Li

TL;DR

This work tackles SFDA for fundus image diagnosis under domain shift, where traditional SFDA methods struggle due to forgetting high-quality target-model predictions and missing ViL's patch-level grounding. It introduces Forgetting-Resistant and Lesion-Aware (FRLA) adaptation, combining a memory-bank–driven forgetting-resistant component with a patch-level, lesion-aware supervision that leverages ViL predictions via dual mutual information losses $L_{dis}$ and $L_{fr}$ and an adaptive patch supervision term $L_{la}$. The approach yields consistent improvements over ViL baselines and SOTA SFDA methods across two cross-domain dataset pairs, with ablations validating memory-bank protection and patch-level guidance. Overall, FRLA demonstrates effective integration of vision-language grounding into SFDA for medical imaging, improving accuracy and lesion localization while preserving reliable target predictions.

Abstract

Source-free domain adaptation (SFDA) aims to adapt a model trained in the source domain to perform well in the target domain, with only unlabeled target domain data and the source model. Taking into account that conventional SFDA methods are inevitably error-prone under domain shift, recently greater attention has been directed to SFDA assisted with off-the-shelf foundation models, e.g., vision-language (ViL) models. However, existing works of leveraging ViL models for SFDA confront two issues: (i) Although mutual information is exploited to consider the joint distribution between the predictions of ViL model and the target model, we argue that the forgetting of some superior predictions of the target model still occurs, as indicated by the decline of the accuracies of certain classes during adaptation; (ii) Prior research disregards the rich, fine-grained knowledge embedded in the ViL model, which offers detailed grounding for fundus image diagnosis. In this paper, we introduce a novel forgetting-resistant and lesion-aware (FRLA) method for SFDA of fundus image diagnosis with ViL model. Specifically, a forgetting-resistant adaptation module explicitly preserves the confident predictions of the target model, and a lesion-aware adaptation module yields patch-wise predictions from ViL model and employs them to help the target model be aware of the lesion areas and leverage the ViL model's fine-grained knowledge. Extensive experiments show that our method not only significantly outperforms the vision-language model, but also achieves consistent improvements over the state-of-the-art methods. Our code will be released.

Forgetting-Resistant and Lesion-Aware Source-Free Domain Adaptive Fundus Image Analysis with Vision-Language Model

TL;DR

This work tackles SFDA for fundus image diagnosis under domain shift, where traditional SFDA methods struggle due to forgetting high-quality target-model predictions and missing ViL's patch-level grounding. It introduces Forgetting-Resistant and Lesion-Aware (FRLA) adaptation, combining a memory-bank–driven forgetting-resistant component with a patch-level, lesion-aware supervision that leverages ViL predictions via dual mutual information losses and and an adaptive patch supervision term . The approach yields consistent improvements over ViL baselines and SOTA SFDA methods across two cross-domain dataset pairs, with ablations validating memory-bank protection and patch-level guidance. Overall, FRLA demonstrates effective integration of vision-language grounding into SFDA for medical imaging, improving accuracy and lesion localization while preserving reliable target predictions.

Abstract

Source-free domain adaptation (SFDA) aims to adapt a model trained in the source domain to perform well in the target domain, with only unlabeled target domain data and the source model. Taking into account that conventional SFDA methods are inevitably error-prone under domain shift, recently greater attention has been directed to SFDA assisted with off-the-shelf foundation models, e.g., vision-language (ViL) models. However, existing works of leveraging ViL models for SFDA confront two issues: (i) Although mutual information is exploited to consider the joint distribution between the predictions of ViL model and the target model, we argue that the forgetting of some superior predictions of the target model still occurs, as indicated by the decline of the accuracies of certain classes during adaptation; (ii) Prior research disregards the rich, fine-grained knowledge embedded in the ViL model, which offers detailed grounding for fundus image diagnosis. In this paper, we introduce a novel forgetting-resistant and lesion-aware (FRLA) method for SFDA of fundus image diagnosis with ViL model. Specifically, a forgetting-resistant adaptation module explicitly preserves the confident predictions of the target model, and a lesion-aware adaptation module yields patch-wise predictions from ViL model and employs them to help the target model be aware of the lesion areas and leverage the ViL model's fine-grained knowledge. Extensive experiments show that our method not only significantly outperforms the vision-language model, but also achieves consistent improvements over the state-of-the-art methods. Our code will be released.
Paper Structure (6 sections, 7 equations, 2 figures, 2 tables)

This paper contains 6 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Forgetting-Resistant Adaptation with Dual Mutual Information Loss
  4. Lesion-Aware Adaptation with Adaptive Patch-level Supervision
  5. Experiments
  6. Conclusion

Figures (2)

  • Figure 1: The framework of our proposed FRLA, which comprises forgetting-resistant adaptation and lesion-aware adaptation. The meanings of the disease abbreviations are given in Sec. \ref{['experiments']}. The text input of ViL model is omitted for simplicity.
  • Figure 2: On the ODIR-to-FIVES adaptation: (a) The evolving dynamics of average accuracy during adaptation for different combinations of the losses. (b) The CAMs generated by the target models trained with our method and our method without $\mathcal{L}_{la}$. The gray boxes mark critical lesions recognized by a senior ophthalmologist.