SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models

TL;DR

SlotFormer presents a Transformer-based autoregressive dynamics model that operates on unsupervised object-centric slots to capture spatio-temporal interactions among multiple objects. By combining a pre-trained object-centric slot extractor with a temporal Transformer, it achieves strong long-horizon video prediction while preserving object identities. Importantly, the unsupervised dynamics knowledge improves downstream tasks such as predictive VQA and goal-conditioned planning, demonstrating the practical value of object-centric world models. The approach advances the ability to reason about future states with minimal supervision and offers a promising avenue for scalable, transferable physical reasoning in vision systems.

Abstract

Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent object-centric models can successfully decompose a scene into objects, modeling their dynamics effectively still remains a challenge. We address this problem by introducing SlotFormer -- a Transformer-based autoregressive model operating on learned object-centric representations. Given a video clip, our approach reasons over object features to model spatio-temporal relationships and predicts accurate future object states. In this paper, we successfully apply SlotFormer to perform video prediction on datasets with complex object interactions. Moreover, the unsupervised SlotFormer's dynamics model can be used to improve the performance on supervised downstream tasks, such as Visual Question Answering (VQA), and goal-conditioned planning. Compared to past works on dynamics modeling, our method achieves significantly better long-term synthesis of object dynamics, while retaining high quality visual generation. Besides, SlotFormer enables VQA models to reason about the future without object-level labels, even outperforming counterparts that use ground-truth annotations. Finally, we show its ability to serve as a world model for model-based planning, which is competitive with methods designed specifically for such tasks.

Figures (14)

• Figure 1: SlotFormer architecture overview. Taking multiple video frames $\{\bm{x}_t\}_{t=1}^T$ as input, we first extract object slots $\{\mathcal{S}_t\}_{t=1}^T$ using the pretrained object-centric model. Then, slots are linearly projected and added with temporal positional encoding. The resulting tokens are fed to the Transformer module to generate future slots $\{\hat{\mathcal{S}}_{T+k}\}_{k=1}^K$ in an autoregressive manner.
• Figure 2: Video dynamics modeling with SlotFormer as a function of future steps. (left) Visual quality of decoded frames measured with LPIPS and (right) the quality of decoded foreground object masks with mIoU.
• Figure 3: Generation results on OBJ3D. Despite higher PSNR, PredRNN and SAVi-dyn produce images with artifacts, while SlotFormer simulates sharp frames and accurate object dynamics.
• Figure 4: Qualitative results on CLEVRER VQA task. To answer the question "Will the green object collide with the purple cylinder?", SlotFormer successfully simulates the first collision between the green and the brown cylinder (t = 13), which leads to the second collision between the target objects (t = 29).
• Figure 5: Qualitative results on PHYRE. The goal is to place a red ball in the first frame, so that the green ball hits the blue object after rollout. We show the per-slot rollout, where SlotFormer is able to decompose the scene into individual objects, and reason their interactions to perform accurate future synthesis.