Improving Segment Anything on the Fly: Auxiliary Online Learning and Adaptive Fusion for Medical Image Segmentation

TL;DR

The paper tackles the challenge of achieving clinically acceptable medical image segmentation with Segment Anything by enabling test-time improvements through Auxiliary Online Learning (AuxOL). AuxOL pairs a small specialist network with the generalist SAM, updates the specialist online using rectified expert segmentations, and fuses outputs with an adaptive mechanism. Experimental results across eight datasets and four modalities show meaningful gains in Dice and Hausdorff Distance for SAM and Medical SAM, including configurations with partial expert feedback and after existing task-specific adaptations. The approach offers a practical, scalable alternative to offline fine-tuning, with potential to reduce expert rectification workload in clinical settings.

Abstract

The current variants of the Segment Anything Model (SAM), which include the original SAM and Medical SAM, still lack the capability to produce sufficiently accurate segmentation for medical images. In medical imaging contexts, it is not uncommon for human experts to rectify segmentations of specific test samples after SAM generates its segmentation predictions. These rectifications typically entail manual or semi-manual corrections employing state-of-the-art annotation tools. Motivated by this process, we introduce a novel approach that leverages the advantages of online machine learning to enhance Segment Anything (SA) during test time. We employ rectified annotations to perform online learning, with the aim of improving the segmentation quality of SA on medical images. To improve the effectiveness and efficiency of online learning when integrated with large-scale vision models like SAM, we propose a new method called Auxiliary Online Learning (AuxOL). AuxOL creates and applies a small auxiliary model (specialist) in conjunction with SAM (generalist), entails adaptive online-batch and adaptive segmentation fusion. Experiments conducted on eight datasets covering four medical imaging modalities validate the effectiveness of the proposed method. Our work proposes and validates a new, practical, and effective approach for enhancing SA on downstream segmentation tasks (e.g., medical image segmentation).

Figures (6)

• Figure 1: Existing and our approaches on utilizing SAM for downstream medical image segmentation.
• Figure 2: An overview of the main steps of our AuxOL with SAM: Improving Segment Anything (SA) for medical images via auxiliary learning in an online learning pipeline.
• Figure 3: AuxOL under varying levels of ground truth supply.
• Figure 4: AuxOL with varying thresholds (in Dice score) to provoke HE rectifications.
• Figure 5: Online learning performance of AuxOL with Already-Adapted SAM.