Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Abstract

Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.

Figures (8)

• Figure 1: Comparison of different MLLM-based segmentation paradigms. Almost all previous methods follow (a) and (b), which rely on the specialist mask decoder. Limited approaches directly predict mask from the MLLM, yet they still require multiple [SEG] tokens for guidance. Our approach in (d) takes advantage of higher resolution pre-compressed features and integrates the accumulated residual features, enabling MLLM-based segmentation without additional specialist decoders and [SEG] tokens.
• Figure 2: The additional branch of pre-compressed image features self-replication for uncompressed features. The compressed features for LLM follow the original process.
• Figure 3: An overview of RFR and RFA operations. The residual features are amplified from the restored compressed features with and without LLM processing. The fusion of restored uncompressed features and the amplified residual features simultaneously achieves higher resolution and fine-grained representations.
• Figure 4: Visualization results on RefCOCO demonstrate the effectiveness of the modules. 'HR' stands for Higher Resolution of uncompressed image features, 'Residual' refers to the use of residual features from the LLM, and 'PUS' represents MLP-with-Pixel-Unshuffle. The bottom row illustrates the resolution of the fused image features and the predicted mask before interpolation to the original image size.
• Figure 5: Representative VQA results comparison.