MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning

TL;DR

MOBIUS introduces a Pareto-efficient, big-to-mobile universal instance segmentation framework that combines a bottleneck pixel decoder with multi-modal fusion, a single-scale transformer decoder, and language-guided uncertainty calibration for adaptive query pruning. The bottleneck encoder condenses multi-scale and text information into a compact representation, reducing pixel-decoder FLOPs by up to 55% and transformer FLOPs by up to 50%, while preserving open-vocabulary performance. A language-guided calibration loss aligns localization confidence with language-based classification scores, enabling inference-time pruning that further halves decoder compute. A unified, single-stage training strategy stabilizes learning across diverse datasets, achieving competitive state-of-the-art performance with substantially reduced training iterations. Overall, MOBIUS sets a new benchmark for efficient universal instance segmentation across devices from mobile to high-end GPUs, demonstrating real-time edge deployment without sacrificing accuracy.

Abstract

Scaling up model size and training data has advanced foundation models for instance-level perception, achieving state-of-the-art in-domain and zero-shot performance across object detection and segmentation. However, their high computational cost limits adoption on resource-constrained platforms. We first examine the limitations of existing architectures in enabling efficient edge deployment without compromising performance. We then introduce MOBIUS, a family of foundation models for universal instance segmentation, designed for Pareto-optimal downscaling to support deployment across devices ranging from high-end accelerators to mobile hardware. To reduce training and inference demands, we propose: (i) a bottleneck pixel decoder for efficient multi-scale and multi-modal fusion, (ii) a language-guided uncertainty calibration loss for adaptive decoder pruning, and (iii) a streamlined, unified training strategy. Unlike efficient baselines that trade accuracy for reduced complexity, MOBIUS reduces pixel and transformer decoder FLOPs by up to 55% and 75%, respectively, while maintaining state-of-the-art performance in just a third of the training iterations. MOBIUS establishes a new benchmark for efficient segmentation on both high-performance computing platforms and mobile devices.

Figures (5)

• Figure 1: Pareto efficiency. The MOBIUS family demonstrates Pareto-efficient downscaling of universal instance segmentation compared to state-of-the-art GLEE. We compare computational requirements (FLOPs) with performance ($AP_{\text{mask}}$) on LVIS-val for big and mobile model sizes. The text encoder fixed cost is omitted.
• Figure 2: Component-wise FLOPs Comparison. We compare MOBIUS to GLEE glee with MaskDINO li2023mask and RT-DETR zhao2024detrs pixel decoders. FLOPs are given as a percentage of an R50 vision encoder (52.4G), excluding the text encoder. Models are profiled at 800×800 resolution. MOBIUS halves all costs while retaining competitive performance wrt. the GLEE-MaskDINO baseline.
• Figure 3: Overview of the MOBIUS framework. The figure illustrates the core components: (i) the novel pixel decoder for efficient multi-scale and multi-modal fusion, and (ii) the transformer decoder with pruning strategy. This design enables Pareto-efficient downscaling for universal instance segmentation.
• Figure 4: Ablation on pruning strategies. We compare the effect of different pruning on the number of decoder FLOPs and the $\rm AP_{mask}$ on COCO-val and LVIS-minival datasets.
• Figure 5: Qualitative results for different instance segmentation supported by our approach. In each row, we show the input image and report the instance segmentation results for (i) category-guided instance segmentation with COCO categories, (ii) category-agnostic instance segmentation, (iii) referring instance segmentation.