Context Matters: Learning Global Semantics via Object-Centric Representation

TL;DR

The paper addresses the lack of semantic guidance in vision transformers by introducing an object-centric masking objective within Masked Image Modeling (MIM). By treating objects as visual words and masking entire objects with a coarse object tokenizer (e.g., SAM), the model learns global context and semantic distributions that support reasoning and in-context capabilities. A two-stage training regime combines the standard MIM loss with an object-balanced loss, yielding improved performance on visual prompting, scene reconstruction, and multimodal VQA tasks when integrated with LLMs like LLaVA and BLIP. The results suggest that object-level encoding enhances semantic understanding and contextual reasoning, offering a plausible path toward stronger vision encoders and tokenizers, with public code and models to follow.

Abstract

Recent advances in language modeling have witnessed the rise of highly desirable emergent capabilities, such as reasoning and in-context learning. However, vision models have yet to exhibit comparable progress in these areas. In this paper, we argue that this gap could stem from the lack of semantic and contextual guidance in current vision transformer (ViT) training schemes, and such a gap can be narrowed through the design of a semantic-grounded objective. Specifically, we notice that individual words in natural language are inherently semantic, and modeling directly on word tokens naturally learns a realistic distribution. In contrast, ViTs rely on spatial patchification, which inevitably lacks semantic information. To bridge this gap, we propose to directly model "object" as the visual equivalence of "word," pushing the model to learn the global context and semantics among visual elements. We investigate our hypotheses via masked image modeling (MIM), a framework where our approach can be readily tested by applying masks to visual objects rather than random patches. Considerable evidence from qualitative and quantitative evaluations reveals a key finding: object-level representation alone helps to learn a real-world distribution, whereas pixel-averaging shortcuts are often learned without it. Moreover, further evaluations with multimodal LLMs (MLLM) on visual question answering (VQA, GQA, ScienceQA) tasks demonstrate the strong reasoning and contextual understanding gained with this simple objective. We hope our study highlights the effectiveness of object-level encoding and provides a plausible direction for developing stronger vision encoders and tokenizers. Code and model will be publicly released. Keywords: Semantic Visual Tokenizer, Vision Reasoning, In-context Learning, Multimodal Reasoning

Figures (11)

• Figure 1: By masking out random patches, current MIM setup encourages a shortcut learning where its generation is entirely based on surrounding pixels with little reference to global context.
• Figure 2: Overall pipeline for Object-Centric MIM.We utilize a pre-trained segmentation model as an object tokenizer to segment the image into coarse object regions. The masked autoencoder is then trained using object-centric masking, and to further enhance the training of the object-centric encoder, we develop object-balanced loss.
• Figure 3: Reconstruction results of 1) scene reconstruction and 2) object-specific reconstruction.
• Figure 4: (a) Visual in-context learning results on various vision tasks. (b) Reconstruction results on toy "context" dataset.
• Figure 5: (A) Object Definition and Context: The representations of objects are learned based on "Color" and "Shape." (B) Minimal Context Reconstruction: Reconstruction performed with minimal context, both with and without object reference.