ChronoDreamer: Action-Conditioned World Model as an Online Simulator for Robotic Planning

TL;DR

ChronoDreamer introduces an action-conditioned world model that jointly predicts future RGB frames, contact maps, and joint angles for contact-rich robotic manipulation. It employs a spatial-temporal transformer with MaskGIT-style masked prediction and renders contact as depth-weighted Gaussian splats, enabling image-native supervision. At inference, rollouts are validated by a vision-language collision judge to reject unsafe actions online, integrating a pragmatic safety filter into planning. Evaluated on the DreamerBench dataset, the approach demonstrates spatially coherent non-contact dynamics and plausible contact predictions, with the LLM-based judge effectively distinguishing collision trajectories. This work advances planning-with-imagination in robotics by coupling dense perceptual predictions with physics- and task-aware validation in real-time.

Abstract

We present ChronoDreamer, an action-conditioned world model for contact-rich robotic manipulation. Given a history of egocentric RGB frames, contact maps, actions, and joint states, ChronoDreamer predicts future video frames, contact distributions, and joint angles via a spatial-temporal transformer trained with MaskGIT-style masked prediction. Contact is encoded as depth-weighted Gaussian splat images that render 3D forces into a camera-aligned format suitable for vision backbones. At inference, predicted rollouts are evaluated by a vision-language model that reasons about collision likelihood, enabling rejection sampling of unsafe actions before execution. We train and evaluate on DreamerBench, a simulation dataset generated with Project Chrono that provides synchronized RGB, contact splat, proprioception, and physics annotations across rigid and deformable object scenarios. Qualitative results demonstrate that the model preserves spatial coherence during non-contact motion and generates plausible contact predictions, while the LLM-based judge distinguishes collision from non-collision trajectories.

Figures (5)

• Figure 1: Data pipeline. Cosmos encoder tokenizes RGB and contact frames into $32 \times 32$ grids. The dynamics model outputs factorized video/contact logits and joint angles.
• Figure 2: ST-Transformer architecture. Control tokens (action + joint embeddings) are concatenated with video tokens and processed by $N$ axial ST-Blocks. Three output heads: joint regression (MSE), contact tokens, and video tokens (factorized CE).
• Figure 3: Spatial coherence in non-contact scenarios (fea_flashlight). Row 1: prompt; Rows 2--3: predicted RGB; Rows 4--5: predicted contact splat. Time progresses left to right.
• Figure 4: Predictions during contact events. Layout as in Fig. \ref{['fig:spatial_coherence']}. Post-contact frames exhibit increased blurriness.
• Figure 5: Consecutive action samples at the same timestep. LLM judge rejects attempt 0, accepts attempt 1.