Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs

TL;DR

This work tackles the problem of adding pixel-level segmentation to Multimodal LLMs without sacrificing their general reasoning abilities. It introduces LENS, a plug-and-play head that remains attached to a frozen MLLM and leverages attention signals to generate keypoints and labels that feed a SAM decoder. LENS achieves state-of-the-art segmentation on benchmarks like ReasonSeg and RefCOCO+ while preserving the MLLM's broader capabilities and dramatically reducing training costs compared to joint fine-tuning approaches. The core idea is to map the MLLM's intrinsic spatial cues into SAM-compatible prompts via a lightweight descriptor model, enabling efficient, on-demand segmentation within a unified vision-language framework.

Abstract

Integrating diverse visual capabilities into a unified model is a significant trend in Multimodal Large Language Models (MLLMs). Among these, the inclusion of segmentation poses a distinct set of challenges. To equip MLLMs with pixel-level segmentation abilities, prevailing methods require finetuning the model to produce specific outputs compatible with a mask decoder. This process typically alters the model's output space and compromises its intrinsic generalization, which undermines the goal of building a unified model. We introduce LENS (Leveraging kEypoiNts for MLLMs' Segmentation), a novel plug-and-play solution. LENS attaches a lightweight, trainable head to a completely frozen MLLM. By refining the spatial cues embedded in attention maps, LENS extracts keypoints and describes them into point-wise features directly compatible with the mask decoder. Extensive experiments validate our approach: LENS achieves segmentation performance competitive with or superior to that of retraining-based methods. Crucially, it does so while fully preserving the MLLM's generalization capabilities, which are significantly degraded by finetuning approaches. As such, the attachable design of LENS establishes an efficient and powerful paradigm for extending MLLMs, paving the way for truly multi-talented, unified models.

Figures (7)

• Figure 1: Conventional architecture for MLLM segmentation vs. LENS.(a) Conventional methods (e.g. LISA lai2024lisa) fine-tune an MLLM for both generation and segmentation tasks, leading to conflicting objectives that undermine the model’s general capabilities and training stability. (b)LENS decouples these roles: a frozen MLLM is dedicated to reasoning, while a lightweight head is trained exclusively for segmentation. The head can be adaptively invoked by the model when needed, allowing the MLLM to serve as a unified vision model capable of handling diverse tasks.
• Figure 2: LENS architecture.(a) Overall architecture built on LLaVA-1.5-7B liu2024improved, consisting of three stages: ① a segmentation head that refines attention and semantic features, ② keypoint extraction and description, and ③ a mask decoder that takes fused keypoint descriptions and coordinates to predict the final mask. Trainable and frozen modules are indicated, and the MLLM runs only in inference mode. (b) Zoom-in view of the attention and keypoint components.
• Figure 3: Showcases of LENS. Attention maps align well with ground-truth regions, where white points mark keypoints inside and red points mark those outside. These results illustrate how LENS links semantics with segmentation.
• Figure S1: The sturcture of the descriptor model.
• Figure S2: Showcases of LENS. The white dots overlaid on the attention maps indicate keypoints that are aligned with the target segmentation regions.