Blackbox Adaptation for Medical Image Segmentation

TL;DR

The paper addresses cross-domain generalization in surgical instrument classification across datasets $CATARACTS$ and $D99$, where domain shift degrades performance. It introduces Barrow Adaptor, an end-to-end unsupervised domain adaptation framework that combines CORAL-based feature alignment with the novel BFAL loss to encourage non-redundant, domain-agnostic representations; BFAL computes a correlation matrix ${\mathbb{C}_1}$ via a projector $P$ and enforces ${\mathbb{L}}_{BFAL}$ to push ${\mathbb{C}_1^{ii}}$ toward 1 and ${\mathbb{C}_1^{ij}}$ toward 0, with the total loss ${\mathbb{L}}_{final} = {\mathbb{L}}_{CE} + \lambda({\mathbb{L}}_{CORAL} + {\mathbb{L}}_{BFAL})$. Empirical results show larger macro-accuracy gains than strong baselines across ResNet50 and ViT backbones, and ablations confirm the contributions of both ${\mathbb{L}}_{CORAL}$ and ${\mathbb{L}}_{BFAL}$, suggesting BFAL's modular applicability to other architectures and UDA methods.

Abstract

In recent years, various large foundation models have been proposed for image segmentation. There models are often trained on large amounts of data corresponding to general computer vision tasks. Hence, these models do not perform well on medical data. There have been some attempts in the literature to perform parameter-efficient finetuning of such foundation models for medical image segmentation. However, these approaches assume that all the parameters of the model are available for adaptation. But, in many cases, these models are released as APIs or blackboxes, with no or limited access to the model parameters and data. In addition, finetuning methods also require a significant amount of compute, which may not be available for the downstream task. At the same time, medical data can't be shared with third-party agents for finetuning due to privacy reasons. To tackle these challenges, we pioneer a blackbox adaptation technique for prompted medical image segmentation, called BAPS. BAPS has two components - (i) An Image-Prompt decoder (IP decoder) module that generates visual prompts given an image and a prompt, and (ii) A Zero Order Optimization (ZOO) Method, called SPSA-GC that is used to update the IP decoder without the need for backpropagating through the foundation model. Thus, our method does not require any knowledge about the foundation model's weights or gradients. We test BAPS on four different modalities and show that our method can improve the original model's performance by around 4%.

Figures (2)

• Figure 1: Domain shift between the CATARACTS dataset (CAT) cataracts and D99 jhu1jhu2 dataset. Results for models trained on one dataset and tested on another show a significant drop in performance, which shows the effect of dataset shift.
• Figure 2: Architecture corresponding to the Barlow Adaptor. Training occurs using pairs of images from the source and target domain. They are fed into the feature extractor, which generates features used for the CORAL loss. Further, a projector network $P$ projects the features into a $P$ dimensional space. These are then used to calculate the Barlow Feature Alignment Loss. One branch from the source features goes into the source classifier network that is used to compute the cross entropy loss with the labeled source data.