Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autoadaptive Medical Segment Anything Model

Tyler Ward, Meredith K. Owen, O'Kira Coleman, Brian Noehren, Abdullah-Al-Zubaer Imran

TL;DR

The paper tackles the challenge of annotation-efficient medical image segmentation by integrating the Segment Anything Model (SAM) with an autoadaptive multitask framework (ADA-SAM). A classification branch generates GradCAM-based prompts to guide SAM, while a gradient-feedback loop allows segmentation gradients to refine classification representations; LoRA enables domain-efficient fine-tuning. A novel SegEx evaluation framework pairs human and LLM observers to assess clinical utility of masks. On thigh MRI data (VL/VM) in patellar instability, ADA-SAM delivers state-of-the-art performance under limited labels, outperforming fully supervised and existing SAM-based semi-supervised baselines, and is complemented by credible qualitative assessments and efficient inference. The work advances practical, annotation-efficient clinical segmentation and introduces SegEx as a principled QA framework for medical mask quality.

Abstract

Medical image segmentation is a key task in the imaging workflow, influencing many image-based decisions. Traditional, fully-supervised segmentation models rely on large amounts of labeled training data, typically obtained through manual annotation, which can be an expensive, time-consuming, and error-prone process. This signals a need for accurate, automatic, and annotation-efficient methods of training these models. We propose ADA-SAM (automated, domain-specific, and adaptive segment anything model), a novel multitask learning framework for medical image segmentation that leverages class activation maps from an auxiliary classifier to guide the predictions of the semi-supervised segmentation branch, which is based on the Segment Anything (SAM) framework. Additionally, our ADA-SAM model employs a novel gradient feedback mechanism to create a learnable connection between the segmentation and classification branches by using the segmentation gradients to guide and improve the classification predictions. We validate ADA-SAM on real-world clinical data collected during rehabilitation trials, and demonstrate that our proposed method outperforms both fully-supervised and semi-supervised baselines by double digits in limited label settings. Our code is available at: https://github.com/tbwa233/ADA-SAM.

Autoadaptive Medical Segment Anything Model

TL;DR

The paper tackles the challenge of annotation-efficient medical image segmentation by integrating the Segment Anything Model (SAM) with an autoadaptive multitask framework (ADA-SAM). A classification branch generates GradCAM-based prompts to guide SAM, while a gradient-feedback loop allows segmentation gradients to refine classification representations; LoRA enables domain-efficient fine-tuning. A novel SegEx evaluation framework pairs human and LLM observers to assess clinical utility of masks. On thigh MRI data (VL/VM) in patellar instability, ADA-SAM delivers state-of-the-art performance under limited labels, outperforming fully supervised and existing SAM-based semi-supervised baselines, and is complemented by credible qualitative assessments and efficient inference. The work advances practical, annotation-efficient clinical segmentation and introduces SegEx as a principled QA framework for medical mask quality.

Abstract

Medical image segmentation is a key task in the imaging workflow, influencing many image-based decisions. Traditional, fully-supervised segmentation models rely on large amounts of labeled training data, typically obtained through manual annotation, which can be an expensive, time-consuming, and error-prone process. This signals a need for accurate, automatic, and annotation-efficient methods of training these models. We propose ADA-SAM (automated, domain-specific, and adaptive segment anything model), a novel multitask learning framework for medical image segmentation that leverages class activation maps from an auxiliary classifier to guide the predictions of the semi-supervised segmentation branch, which is based on the Segment Anything (SAM) framework. Additionally, our ADA-SAM model employs a novel gradient feedback mechanism to create a learnable connection between the segmentation and classification branches by using the segmentation gradients to guide and improve the classification predictions. We validate ADA-SAM on real-world clinical data collected during rehabilitation trials, and demonstrate that our proposed method outperforms both fully-supervised and semi-supervised baselines by double digits in limited label settings. Our code is available at: https://github.com/tbwa233/ADA-SAM.

Paper Structure

This paper contains 8 sections, 2 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. ADA-SAM
  4. SegEx: Expert-driven Segmentation Assessment
  5. Experimental Evaluation
  6. Implementation Details
  7. Results and Discussion
  8. Conclusions

Figures (2)

  • Figure 1: Overview of the ADA-SAM framework. ADA-SAM is an autoadaptive, multitask learning model that utilizes a classification branch to enable automated prompt generation for the SAM-based segmentation branch. ADA-SAM utilizes a gradient feedback mechanism, where segmentation gradients refine classification representations, serving as task guidance for the segmentation. LoRA is applied to SAM’s transformer encoder for parameter-efficient domain adaptation.
  • Figure 2: Qualitative comparison demonstrates the superiority of semi-supervised ADA-SAM over the best-performing fully supervised model in segmenting VL and VM Color code: Green - ground truth VM, Blue - ground truth VL, Red - predicted contour.