Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM

TL;DR

This work introduces Slot Q-Former, an object-centric visual tokenizer that converts images into $N=32$ discrete tokens of dimension $D=768$ via slot attention and residual vector quantization, enabling images to be processed as a second language by LLMs. It trains Slot Q-Former in two stages to produce continuous embeddings and then quantize them for discrete multimodal input/output, with image-text alignment losses guiding cross-modal integration. Integrated into a Vicuna-7B-based Slot-MLLM and further optimized with LoRA and multimodal instruction tuning, the approach achieves strong object-centric understanding and generation, notably in localized image editing, outperforming baselines like SEED-LLaMA and LaViT under similar conditions. This first large-scale demonstration of Slot Attention with LLMs on real-world images highlights the practicality and advantages of object-centric tokenization for unified multimodal reasoning and generation.

Abstract

Recently, multimodal large language models (MLLMs) have emerged as a key approach in achieving artificial general intelligence. In particular, vision-language MLLMs have been developed to generate not only text but also visual outputs from multimodal inputs. This advancement requires efficient image tokens that LLMs can process effectively both in input and output. However, existing image tokenization methods for MLLMs typically capture only global abstract concepts or uniformly segmented image patches, restricting MLLMs' capability to effectively understand or generate detailed visual content, particularly at the object level. To address this limitation, we propose an object-centric visual tokenizer based on Slot Attention specifically for MLLMs. In particular, based on the Q-Former encoder, diffusion decoder, and residual vector quantization, our proposed discretized slot tokens can encode local visual details while maintaining high-level semantics, and also align with textual data to be integrated seamlessly within a unified next-token prediction framework of LLMs. The resulting Slot-MLLM demonstrates significant performance improvements over baselines with previous visual tokenizers across various vision-language tasks that entail local detailed comprehension and generation. Notably, this work is the first demonstration of the feasibility of object-centric slot attention performed with MLLMs and in-the-wild natural images.

Figures (6)

• Figure 1: Overview of Slot-MLLM. Slot-MLLM employs Slot Q-Former with Slot Attention to encode images into discrete, object-centric visual tokens. These tokens are treated as a second language within the model, enabling unified autoregressive modeling across multimodal tasks such as image editing (IT2I), visual question answering (I2T), and image generation (T2I).
• Figure 2: Our tokenizer comprises three main modules: Slot Q-Former, Vector Quantizer, and Visual Decoder. Training occurs in two stages: modules trained in each stage are indicated, and modules exclusive to the second stage—marked in red—include the Vector Quantizer for discrete tokenization.
• Figure 3: Qualitative results of visual tokenizers. This figure visualizes the reconstructed images generated by various tokenizers, along with the corresponding image regions attended by each token. Slot Q-Former effectively captures object-centric representations by aligning token attention with distinct objects within the image, and can reconstruct images that preserve the original object structures and compositions.
• Figure 4: Qualitative results on image editing tasks. Slot-MLLM effectively modifies specific objects described by text prompts while preserving the overall image composition and context.
• Figure 5: Qualitative results for variations of Slot Q-Former. When using the object-level image-text contrastive loss, multiple slot tokens are sometimes assigned to a single object. In contrast, models trained without the image-text alignment loss exhibit better image reconstruction quality but produce attention maps that are less object-centric. These observations require further validation.