STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation

TL;DR

STORM tackles the gap between dense foundation-model features and task-oriented, object-centric control in robotics by introducing a two-phase, slot-based adaptation on top of frozen visual backbones. It first performs visual–semantic slot pretraining using text prompts and CLIP guidance to produce stable, interpretable object slots, then couples these slots with a downstream Transformer policy in a joint but gradient-detached manner to preserve semantics while aligning with manipulation objectives. The approach uses a compact pipeline with losses $L_{recons}$, $L_{sem}$, and $L_{pen}$, yielding $L_{Overall} = L_{recons} + L_{sem} + L_{pen}$ for the visual module, and a separate imitation-learning objective for the policy alongside a Slot–Slot contrastive term for temporal consistency. Empirically, STORM improves robustness to visual distractors and generalization in simulated robotic manipulation on MetaWorld and LIBERO benchmarks, outperforming frozen backbone baselines and end-to-end OCRL, with ablations highlighting the necessity of multi-phase learning and semantic grounding. The findings indicate that multi-phase adaptation is an efficient strategy to transform generic foundation-model features into task-aware, object-centric representations suitable for reliable robotic control, enabling better open-vocabulary grounding and resilience in cluttered environments.

Abstract

Visual foundation models provide strong perceptual features for robotics, but their dense representations lack explicit object-level structure, limiting robustness and contractility in manipulation tasks. We propose STORM (Slot-based Task-aware Object-centric Representation for robotic Manipulation), a lightweight object-centric adaptation module that augments frozen visual foundation models with a small set of semantic-aware slots for robotic manipulation. Rather than retraining large backbones, STORM employs a multi-phase training strategy: object-centric slots are first stabilized through visual--semantic pretraining using language embeddings, then jointly adapted with a downstream manipulation policy. This staged learning prevents degenerate slot formation and preserves semantic consistency while aligning perception with task objectives. Experiments on object discovery benchmarks and simulated manipulation tasks show that STORM improves generalization to visual distractors, and control performance compared to directly using frozen foundation model features or training object-centric representations end-to-end. Our results highlight multi-phase adaptation as an efficient mechanism for transforming generic foundation model features into task-aware object-centric representations for robotic control.

Figures (3)

• Figure 1: Overview of STORM. STORM follows a two-stage training to produce task-aware object-centric representations for robotic control. (Step 1) Semantic learning: Frozen DINOv2 features are aggregated by a Slot-Attention module conditioned on noun embeddings extracted from text prompts using a frozen CLIP-text encoder. Reconstruction and contrastive losses encourage stable, semantically grounded slot formation. (Step 2) Dynamic task alignment: The pretrained object-centric module extracts slots from camera observations, which are combined with task embeddings, robot proprioception, and a learnable [ACT] token and processed by a Transformer decoder policy. An GMM action head predicts the next action from the [ACT] token.
• Figure 2: Robotics environments visualization. Examples of the simulated environments used in our experiments: MetaWorld (top row) and LIBERO (bottom row).
• Figure 3: Comparison of slot masks. Visualization of slot masks obtained from training from scratch (top row) versus our two-step training with adaptation (bottom row). Our setup provides much sharper masks with a focus on task-specif objects: the robot arm, the gripper, the drawer's handle and finally the drawer body; while the training from scratch generates noisy masks with no proper focus.