SAMTok: Representing Any Mask with Two Words

TL;DR

SAMTok reimagines region masks as a two-token language by introducing a SAM-based mask encoder and a two-step residual vector quantizer, yielding two discrete tokens per region mask. This enables base multimodal LLMs to learn pixel-level capabilities through standard next-token prediction and a purely textual reinforcement signal, without modifying model architectures or losses. Trained on 209M masks and paired with ~5M mask–text data, SAMTok demonstrates state-of-the-art or competitive performance across region captioning, VQA, grounded grounding, referring segmentation, scene graphs, and interactive segmentation, with notable gains from text-based RL on GRES and GCG. The approach offers a scalable, unified framework for pixel-wise reasoning in large language models, unlocking robust region-aware capabilities with minimal engineering changes.

Abstract

Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask using these tokens with high fidelity. By treating masks as new language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.

Figures (11)

• Figure 1: SAMTok provides a simple yet unified mask-token interface for MLLMs. (Left) SAMTok compresses region masks into two discrete tokens and faithfully reconstructs them across diverse visual domains. (Middle) Injecting these mask tokens into MLLMs enables a wide range of region-level mask generation and understanding tasks. (Right) The text-based representation of region masks allows a purely textual answer-matching reward for the GRPO of the mask generation task.
• Figure 2: Our SAMTok architecture (Left) and mask reconstruction examples (Right). SAMTok has a encoder $f_{\text{enc}}$, a vector quantizer with codebook $\mathcal{C}$, and a decoder $f_{\text{dec}}$. Both $f_{\text{enc}}$ and $f_{\text{dec}}$ are instantiated with a SAM model, which includes an image backbone $f_{\text{img}}$, a prompt encoder $f_{\text{prm}}$, and a mask decoder $f_{\text{msk}}$. Given an input image $\mathcal{I}$ and a region $\mathcal{M}$ (e.g., the area outlined in purple), the SAMTok encoder $f_{\text{enc}}$ first encodes the 2D mask into a mask embedding $\mathbf{z}$, then performs two-stage quantization to obtain a discrete mask embedding $[\mathbf{e}_1, \mathbf{e}_2]$. The SAMTok decoder $f_{\text{dec}}$ reconstructs the 2D mask $\hat{\mathcal{M}}$ from the original image and the region’s discrete mask embeddings.
• Figure 3: Unified mask-token interface for MLLMs. For the mask understanding task (left to right), SAMTok first tokenizes region masks into quantization codes, then formats them into mask words, which are used in the MLLM prompt to refer to the corresponding image regions. For the mask generation task (right-to-left), the MLLM first produces mask words according to the instruction, then maps them to quantization codes, after which SAMTok reconstructs the 2D masks.
• Figure 4: Overview of dataset used to train the SAMTok (left) and MLLM (right). We use 209M masks to train SAMTok, and 5M conversations to fine tune MLLMs.
• Figure 5: Qualitative examples of SAMTok on diverse downstream tasks, including panoptic scene graph generation (top-left), generalized referring expression segmentation (bottom-left), region captioning (top-right), and grounded conversation generation (bottom-right).