Seg-VAR: Image Segmentation with Visual Autoregressive Modeling

TL;DR

Seg-VAR reframes segmentation as a conditional autoregressive mask generation problem by learning discrete seglat tokens via a location-aware color mapping and recovering masks through a hierarchical transformer decoder. The method combines an image encoder that provides latent priors, a seglat latent encoder/decoder, and a latent encoder/decoder to align image-derived latents with seglat distributions using an ELBO objective with a KL term. A three-stage training procedure—seglat learning, latent learning, and image-prior alignment—enables cross-task performance, achieving state-of-the-art results on COCO, Cityscapes, and ADE20K across semantic, instance, and panoptic segmentation. Empirical results and ablations show the value of spatially aware latents and multi-scale autoregressive decoding for precise, context-aware segmentation. This work suggests a broadly applicable pathway for integrating autoregressive reasoning into spatial vision systems and opens avenues for future extensions to video and other structured perception tasks.

Abstract

While visual autoregressive modeling (VAR) strategies have shed light on image generation with the autoregressive models, their potential for segmentation, a task that requires precise low-level spatial perception, remains unexplored. Inspired by the multi-scale modeling of classic Mask2Former-based models, we propose Seg-VAR, a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem. This is achieved by replacing the discriminative learning with the latent learning process. Specifically, our method incorporates three core components: (1) an image encoder generating latent priors from input images, (2) a spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens using a location-sensitive color mapping to distinguish instances, and (3) a decoder reconstructing masks from these latents. A multi-stage training strategy is introduced: first learning seglat representations via image-seglat joint training, then refining latent transformations, and finally aligning image-encoder-derived latents with seglat distributions. Experiments show Seg-VAR outperforms previous discriminative and generative methods on various segmentation tasks and validation benchmarks. By framing segmentation as a sequential hierarchical prediction task, Seg-VAR opens new avenues for integrating autoregressive reasoning into spatial-aware vision systems. Code will be available at https://github.com/rkzheng99/Seg-VAR.

Figures (4)

• Figure 1: Our Seg-VAR is a visual autoregressive model that is designed for generic image segmentation. Different from (a) traditional discriminative segmentation models and (b) diffusion-based generative models that mainly take input image as a condition, our Seg-VAR rethinks segmentation as a sequence modeling task by encoding the input image to a latent distribution and generating the masks hierarchically.
• Figure 2: Illustration of the latent and seglat learning ($q_\phi, p_\theta$) of proposed Seg-VAR. We first jointly model the seglat and image during training in the seglat encoder and decoder (red module). Then, with the well-trained encoder and decoder, we try to optimize the latent encoder and decoder (green module). Worth mentioning, we use different color in the binary segmentation mask to highlight different instances of the image.
• Figure 3: Illustration of the latent learning ($p_\psi$) of proposed Seg-VAR. In order to learn the latent representations, Seg-VAR optimizes the image encoder while freezing the seglat encoder. We also introduce the latent loss to minimize the divergence between two latent distributions for inference.
• Figure 4: Illustration of the inference stage. The latent distribution generated by the image encoder is fed to the seglat decoder to generate the predicted seglat, and then finally generates the final prediction.