Spotlighting Task-Relevant Features: Object-Centric Representations for Better Generalization in Robotic Manipulation

TL;DR

This paper tackles the generalization gap in visuomotor robot policies by rethinking visual representations. It introduces Slot-based Object-Centric Representations (SOCR) and an object-centric variant of the DINOSAUR architecture (DINOSAUR* and DINOSAUR-Rob*) to produce a finite set of object slots from dense features, which are then integrated into a unified policy framework with frozen encoders. Through an open-source evaluation framework, it benchmarks seven visual representations across simulated and real-world robotic tasks, showing that SOCRs outperform global and dense features in learning efficiency and robustness, especially under visual distribution shifts; robotic pretraining further enhances performance. The results suggest that object-centric, compositional representations enable more reliable generalization in dynamic, real-world environments, providing a promising direction for perceptual design in robotics and beyond.

Abstract

The generalization capabilities of robotic manipulation policies are heavily influenced by the choice of visual representations. Existing approaches typically rely on representations extracted from pre-trained encoders, using two dominant types of features: global features, which summarize an entire image via a single pooled vector, and dense features, which preserve a patch-wise embedding from the final encoder layer. While widely used, both feature types mix task-relevant and irrelevant information, leading to poor generalization under distribution shifts, such as changes in lighting, textures, or the presence of distractors. In this work, we explore an intermediate structured alternative: Slot-Based Object-Centric Representations (SBOCR), which group dense features into a finite set of object-like entities. This representation permits to naturally reduce the noise provided to the robotic manipulation policy while keeping enough information to efficiently perform the task. We benchmark a range of global and dense representations against intermediate slot-based representations, across a suite of simulated and real-world manipulation tasks ranging from simple to complex. We evaluate their generalization under diverse visual conditions, including changes in lighting, texture, and the presence of distractors. Our findings reveal that SBOCR-based policies outperform dense and global representation-based policies in generalization settings, even without task-specific pretraining. These insights suggest that SBOCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.

Figures (4)

• Figure 1: Overview of visual representations. (Left) We use a set of pre-trained visual models with different latent-space structures: global, dense, and object-centric. Dense representations are extracted from one of the encoder's layers (CNN or ViT) before linear projection, while global representations are obtained after pooling operations. Slot-based object-centric representations emerge from an additional Slot-Attention layer that binds every dense feature to a finite set of slots. (Right) We visualize how each representation attends to different parts of the image. Most methods focus narrowly and may be distracted by irrelevant regions. In contrast, object-centric representations (DINOSAUR*) attend to multiple parts, naturally separating task-relevant from irrelevant information. A broader analysis appears in Section \ref{['sec:focus']}.
• Figure 2: Overview of the robotic manipulation policy architecture. We use a pre-trained visual model to extract visual features from raw images. These features are then combined with other modalities (e.g., language instructions, proprioception) in an observation trunk with an additional action token. Finally, a policy head predicts the next action given the additional token. We experiment with different types of visual models that produce global, dense or object-centric representations.
• Figure 3: Overview of real-world setup. We evaluate the different visual models on a Franka robotic arm on four tabletop manipulation tasks (From left to right): Stacking bowls into a pan, Opening a drawer placing a screwdriver inside and closing the drawer, Putting cans into a bin and Placing plates into a dish rack.
• Figure 4: Overall success rate on in-domain and generalization scenarios. Mean success rate over all tasks for each visual model on MetaWorld (left), LIBERO (middle) and Real robot using Franka (right). Green dot: in-domain performance, Orange dot: average performance over all generalization scenarios (distractors, novel textures, lighting changes). Red number: relative drop in performance from in-domain to generalization settings.