PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

TL;DR

PlaySlot presents a controllable, object-centric world model that learns from unlabeled videos by decomposing scenes into slots, inferring inverse dynamics to form latent actions, and forecasting future object states with a transformer-based predictor. The framework combines SAVi-based object decomposition, a hybrid discrete-continuous latent action space, and a conditional object-centric predictor to produce accurate, interpretable future frames and trajectories. It demonstrates strong performance on diverse robotic datasets, enabling sample-efficient learning of robot behaviors from unlabeled demonstrations and showing promising applicability to real-world robotics data. While effective, it notes limitations in decomposition complexity and action disentanglement and outlines avenues for scaling to more complex scenes with richer action factors.

Abstract

Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on videos and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available on https://play-slot.github.io/PlaySlot/.

Figures (21)

• Figure 1: PlaySlot parses an image $\textbf{X}_{1}$ into its object components $\textbf{S}_{1}$. It then predicts multiple future object states and frames with an object-centric video prediction module (cOCVP) conditioned on latent actions $\boldsymbol{Z}$, which can be inferred from a reference video with our $\text{InvDyn}$ module, provided as input, or generated by a learned action policy.
• Figure 2: Overview of PlaySlot training and inference processes. (a) PlaySlot is trained given unlabeled video sequences by inferring object representations $\textbf{S}_{t}$ and latent actions $\mathbf{\hat{z}}_{t}$, and using these representations to autoregressively forecast future video frames and object states. (b) PlaySlot autoregressively forecasts future frames conditioned on a single frame $\textbf{X}_{1}$ and latent actions $\mathbf{\hat{z}}$, which can be inferred from observations, provided by a user, or output by a learned action policy.
• Figure 3: Qualitative comparison on (a) ButtonPress and (b) BlockPush datasets. $\text{PlaySlot}$ accurately predicts the scene dynamics, whereas baselines fail to predict object interactions, leading to blurriness and disappearing objects.
• Figure 4: Quantitative results on the GridShapes dataset with different numbers of objects. PlaySlot outperforms the baselines, particularly for a higher number of objects.
• Figure 5: PlaySlot predictions given different latent actions, including inferred inverse dynamics and three action prototypes. PlaySlot learns accurate object-centric representations and semantically consistent action prototypes.