MultiModal Action Conditioned Video Generation

TL;DR

This work addresses the need for fine-grained, real-time control in video-based world models for household robotics by introducing multisensory action conditioning. It develops a multimodal feature learning framework that preserves modality-specific information while aligning signals in a shared space, and introduces a latent interaction regularization (y'_t) to capture context-consequence dynamics, including a relaxed hyperplane formulation. A diffusion-based video simulator (I2VGen backbone) is conditioned on the learned multisensory action features, trained with a weighted loss that combines diffusion prediction, self-supervised reconstruction, and action-trajectory regularization. Empirical results on ActionSense demonstrate that multisensory conditioning reduces prediction error and temporal drift, robustly handles test-time missing modalities, and enables downstream tasks such as policy optimization and multisensory action planning, underscoring the practical potential for real-world robotic digital twins and fine-grained control systems.

Abstract

Current video models fail as world model as they lack fine-graiend control. General-purpose household robots require real-time fine motor control to handle delicate tasks and urgent situations. In this work, we introduce fine-grained multimodal actions to capture such precise control. We consider senses of proprioception, kinesthesia, force haptics, and muscle activation. Such multimodal senses naturally enables fine-grained interactions that are difficult to simulate with text-conditioned generative models. To effectively simulate fine-grained multisensory actions, we develop a feature learning paradigm that aligns these modalities while preserving the unique information each modality provides. We further propose a regularization scheme to enhance causality of the action trajectory features in representing intricate interaction dynamics. Experiments show that incorporating multimodal senses improves simulation accuracy and reduces temporal drift. Extensive ablation studies and downstream applications demonstrate the effectiveness and practicality of our work.

Figures (21)

• Figure 1: Overview. We introduce a new task for fine-grained control of video generative model using multisensory interaction signals.
• Figure 2: Overview. We focus on learning effective multimodal action representations and propose a generative simulation method.
• Figure 3: Latent Interaction
• Figure 4: Temporal drift. LPIPS per frame.
• Figure 5: Comparison to Unimodal Simulation. We compare our proposed multisensory conditioning to unimodal conditioning, including text and each action sensory modality. The first four frames are the context frames, and the last eight frames are predictions by each method.