AutoMiSeg: Automatic Medical Image Segmentation via Test-Time Adaptation of Foundation Models
Xingjian Li, Qifeng Wu, Adithya S. Ubaradka, Yiran Ding, Colleen Que, Runmin Jiang, Jianhua Xing, Tianyang Wang, Min Xu
TL;DR
AutoMiSeg tackles annotation-free, zero-shot medical image segmentation by composing a grounding module, visual-prompt boosting, and a promptable segmentation model, all guided by Learnable Test-time Adaptors optimized via Bayesian Optimization with a proxy validator. It leverages off-the-shelf foundation models for grounding and segmentation, enabling automatic task-specific prompts without training. Across seven diverse datasets, AutoMiSeg achieves a Dice improvement from $42.53$ to $71.81$, a $69\%$ relative gain, and reaches competitive performance with weak-prompt interactive models, highlighting a scalable path toward annotation-efficient clinical image analysis. The modular design and AutoML-driven adaptation make it a promising platform for extending zero-shot segmentation to broader medical imaging domains.
Abstract
Medical image segmentation is vital for clinical diagnosis, yet current deep learning methods often demand extensive expert effort, i.e., either through annotating large training datasets or providing prompts at inference time for each new case. This paper introduces a zero-shot and automatic segmentation pipeline that combines off-the-shelf vision-language and segmentation foundation models. Given a medical image and a task definition (e.g., "segment the optic disc in an eye fundus image"), our method uses a grounding model to generate an initial bounding box, followed by a visual prompt boosting module that enhance the prompts, which are then processed by a promptable segmentation model to produce the final mask. To address the challenges of domain gap and result verification, we introduce a test-time adaptation framework featuring a set of learnable adaptors that align the medical inputs with foundation model representations. Its hyperparameters are optimized via Bayesian Optimization, guided by a proxy validation model without requiring ground-truth labels. Our pipeline offers an annotation-efficient and scalable solution for zero-shot medical image segmentation across diverse tasks. Our pipeline is evaluated on seven diverse medical imaging datasets and shows promising results. By proper decomposition and test-time adaptation, our fully automatic pipeline not only substantially surpasses the previously best-performing method, yielding a 69\% relative improvement in accuracy (Dice Score from 42.53 to 71.81), but also performs competitively with weakly-prompted interactive foundation models.