Object-Centric Representations Improve Policy Generalization in Robot Manipulation

TL;DR

This work investigates object-centric representations (OCR) as a structured alternative to traditional global or dense visual encoders for robotic manipulation. By benchmarking seven pretrained encoders, including OCR variants DINOSAUR and VIDEOSAUR, across simulated and real-world tasks, the study finds OCR-based policies generalize more robustly under appearance changes and distractors, even without task-specific pretraining. A unified framework combines OCR-based vision with imitation-learning policies, trained on diverse robotic video data to align representations with manipulation dynamics. The results support OCR as a scalable approach for improving sim-to-real transfer and robustness in dynamic robotic environments.

Abstract

Visual representations are central to the learning and generalization capabilities of robotic manipulation policies. While existing methods rely on global or dense features, such representations often entangle task-relevant and irrelevant scene information, limiting robustness under distribution shifts. In this work, we investigate object-centric representations (OCR) as a structured alternative that segments visual input into a finished set of entities, introducing inductive biases that align more naturally with manipulation tasks. We benchmark a range of visual encoders-object-centric, global and dense methods-across a suite of simulated and real-world manipulation tasks ranging from simple to complex, and evaluate their generalization under diverse visual conditions including changes in lighting, texture, and the presence of distractors. Our findings reveal that OCR-based policies outperform dense and global representations in generalization settings, even without task-specific pretraining. These insights suggest that OCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.

Figures (8)

• Figure 1: Overall architecture. We use a set of pre-trained visual models with different structures of latent space - global, dense and object-centric - (a) as input to a policy model for robotic manipulation learning (b). We showcase the benefits of Object-Centric Representations (VIDEOSAUR, DINOSAUR) over different visual models on the final performance of policies and the generalization capabilities in simulation and real-world scenarios (c). DINOSAUR* and VIDEOSAUR* are our versions of the OCR models with the attention module trained using a mixture of robot data.
• Figure 2: Overview environments. We evaluate the different visual models in two simulation (a: MetaWorld, b: LIBERO) and one real-world environments (c).
• Figure 3: Overall success rate. Mean success rate over all tasks for each visual model on the three environments, e.g., MetaWorld (left), LIBERO (middle) and Real using LeRobot (right). DINOSAUR* and VIDEOSAUR* have been pretrained over robot data mixture.
• Figure 4: Real world setup. Our setup is based on the LeRobot library. We use a SO-100 arm on a tabletop environment with two realsense cameras, one with a top overview of the scene and a second with a side view.
• Figure 5: Overview tasks real-world. From left to right: Banana bowl, Open drawer, Pick coffee, Close drawer, Fold bag.