Towards Pixel-Level VLM Perception via Simple Points Prediction

TL;DR

This work interrogates whether pixel-level segmentation can emerge from a standard Multimodal Large Language Model without decoder-based heads. It introduces SimpleSeg, which reframes segmentation as predicting a contour as a sequence of points in the language space and trains with a two-stage SFT→RL pipeline using IoU-based rewards. Key contributions include representing masks as boundary point trajectories, a unified text-based query interface, an automated annotation pipeline, and substantial empirical evidence that SimpleSeg achieves competitive or superior results on RES and REC benchmarks while remaining decoder-free. The findings suggest that fine-grained spatial perception can be an emergent property of MLLMs, offering a simpler, more general framework for pixel-level perception and potentially guiding the development of unified, versatile vision-language models.

Abstract

We present SimpleSeg, a strikingly simple yet highly effective approach to endow Multimodal Large Language Models (MLLMs) with native pixel-level perception. Our method reframes segmentation as a simple sequence generation problem: the model directly predicts sequences of points (textual coordinates) delineating object boundaries, entirely within its language space. To achieve high fidelity, we introduce a two-stage SF$\to$RL training pipeline, where Reinforcement Learning with an IoU-based reward refines the point sequences to accurately match ground-truth contours. We find that the standard MLLM architecture possesses a strong, inherent capacity for low-level perception that can be unlocked without any specialized architecture. On segmentation benchmarks, SimpleSeg achieves performance that is comparable to, and often surpasses, methods relying on complex, task-specific designs. This work lays out that precise spatial understanding can emerge from simple point prediction, challenging the prevailing need for auxiliary components and paving the way for more unified and capable VLMs. Homepage: https://simpleseg.github.io/

Figures (16)

• Figure 1: In this work, we explore the limits of MLLM pixel-level perception by predicting the next point in a contour with the simplest approach possible. Without introducing any complex architectures or special patterns, we show how even minimalistic point prediction can achieve effective segmentation at the pixel level.
• Figure 2: Segmentation results of SimpleSeg on natural and non-natural images. These examples highlight the model's excellent generalization, showing its precise pixel-level perception is not confined to real-world objects. The model successfully segments targets from natural photographs (the lightning) and performs with equal precision on various forms of "in-screen" or digitally generated content, including anime, data charts, and infographics.
• Figure 3: Overview of our data annotation pipeline, which incorporates modules for object detection, mask segmentation, points conversion, and instance caption.
• Figure 4: The relationship between the sequence length and performance under the control of the point density parameter $\epsilon$.
• Figure 5: gIoU score with different rewards.