ARGenSeg: Image Segmentation with Autoregressive Image Generation Model

TL;DR

ARGenSeg tackles the challenge of integrating dense image segmentation into multimodal large language models by reframing segmentation as autoregressive image generation. It uses a frozen VQ-VAE visual tokenizer and a unified MLLM that directly predicts image tokens, augmented with a next-scale generation strategy to enable coarse-to-fine, parallel token generation for efficiency. The approach achieves state-of-the-art results on referring segmentation and generalized RES benchmarks, while maintaining multimodal understanding and offering extensions to interactive segmentation and image generation with substantial speedups over sequential methods. This work demonstrates a viable path toward unified, pixel-accurate visual understanding within a single, generation-capable model, reducing reliance on task-specific segmentation heads and enabling broader applications in AGI-like vision-language systems.

Abstract

We propose a novel AutoRegressive Generation-based paradigm for image Segmentation (ARGenSeg), achieving multimodal understanding and pixel-level perception within a unified framework. Prior works integrating image segmentation into multimodal large language models (MLLMs) typically employ either boundary points representation or dedicated segmentation heads. These methods rely on discrete representations or semantic prompts fed into task-specific decoders, which limits the ability of the MLLM to capture fine-grained visual details. To address these challenges, we introduce a segmentation framework for MLLM based on image generation, which naturally produces dense masks for target objects. We leverage MLLM to output visual tokens and detokenize them into images using an universal VQ-VAE, making the segmentation fully dependent on the pixel-level understanding of the MLLM. To reduce inference latency, we employ a next-scale-prediction strategy to generate required visual tokens in parallel. Extensive experiments demonstrate that our method surpasses prior state-of-the-art approaches on multiple segmentation datasets with a remarkable boost in inference speed, while maintaining strong understanding capabilities.

Figures (7)

• Figure 1: ARGenSeg is a unified framework for visual understanding, segmentation, and generation. It supports semantic, instance, interactive, and zero-shot reasoning segmentation, as well as anomaly detection, by leveraging strong visual understanding capabilities.
• Figure 2: The architecture of ARGenSeg and its training and inference procedures. Left: ARGenSeg integrates image segmentation into the MLLM via an autoregressive image generation paradigm. A unified classification prediction head is used to generate both text and visual tokens. Right: Visual tokens are generated in parallel using the next-scale prediction strategy. During training, a VAE encoder is used to construct supervision for cross-entropy loss. During inference, the VAE decoder reconstructs the image from the predicted visual tokens. [S]/[E] denotes <gen_start>/<gen_end>.
• Figure 3: Multi-scale generation process of the segmentation mask. The model first localizes the target object and then progressively refines its boundaries.
• Figure 4: Top: Visualization of interactive segmentation. Points and scribbles are provided as visual prompts, while bounding boxes are input via text. Bottom: Visualization results of instruction-based image generation. The model is trained on image generation data for only $50k$ iterations.
• Figure 5: Visualization of using different segmentation instructions in the same image.