TextOCVP: Object-Centric Video Prediction with Language Guidance

TL;DR

TextOCVP presents a text-guided, object-centric video prediction framework that parses scenes into slots and uses a text-conditioned transformer to forecast future object states and frames. By grounding predictions in per-object representations and language guidance, the model achieves controllable, interpretable, and robust long-horizon predictions, outperforming existing text-conditioned VP baselines on CATER and CLIPort. The approach demonstrates that structured latent spaces paired with language cues improve generalization to novel object counts and colors, and supports fine-grained control over predicted futures, with potential impact on robotic planning and reasoning. Overall, TextOCVP offers a scalable, interpretable pathway toward language-guided, object-centric manipulation in simulated environments and beyond.

Abstract

Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments. Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and predicting future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability in robotics. To address these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. TextOCVP parses an observed scene into object representations, called slots, and utilizes a text-conditioned transformer predictor to forecast future object states and video frames. Our approach jointly models object dynamics and interactions while incorporating textual guidance, enabling accurate and controllable predictions. TextOCVP's structured latent space offers a more precise control of the forecasting process, outperforming several video prediction baselines on two datasets. Additionally, we show that structured object-centric representations provide superior robustness to novel scene configurations, as well as improved controllability and interpretability, enabling more precise and understandable predictions. Videos and code are available at https://play-slot.github.io/TextOCVP.

Figures (18)

• Figure 1: Overview of TextOCVP. (a) Our model parses a reference frame $\textbf{X}_{1}$ into its object components $\textbf{S}_{1}$. (b) Our TextOCVP predictor jointly models object dynamics and interactions guided by text, generating future object states and frames that align with the provided textual instructions.
• Figure 2: Overview of TextOCVP. Our model parses the reference frame $\textbf{X}_{1}$ into object representations $\textbf{S}_{1}$. The text-conditioned object-centric predictor models object dynamics, incorporating information from the description $\mathbf{C}$ to predict future object states $\hat{\mathbf{S}}_{2:T+1}$, which can be decoded into video frames $\hat{\mathbf{X}}_{2:{T+1}}$.
• Figure 3: Overview of TextOCVP's text-conditioned object-centric video predictor transformer.
• Figure 4: Qualitative comparison of TextOCVP and baseline methods on text-guided video prediction.
• Figure 5: Visualization of text-to-slot attention in TextOCVP. (a) Slots attend to relevant text tokens, grounding objects to their described motions. (b) Distinct attention heads focus on complementary textual cues, such as object attributes and actions.