Unified Multimodal Diffusion Forcing for Forceful Manipulation

TL;DR

MDF tackles the problem of learning from multimodal robotic trajectories by introducing a 2D Time–Modality Noise Level Matrix $\mathbf{K}$ that enables continuous, partial masking across time and modalities. The approach combines a diffusion-based point-cloud autoencoder with a latent diffusion transformer to model temporal and cross-modal dependencies, and supports flexible inference-time tasks including policy, planning, dynamics modeling, state estimation, and anomaly detection. Empirical results across simulated and real-world contact-rich tasks show MDF achieves competitive or superior performance to specialized baselines, while offering robustness to noisy observations and the ability to adapt history lengths and input modalities. This framework promises versatile, robust multimodal reasoning for manipulation, with practical impact on real-world robotic systems under occlusion and sensory noise.

Abstract

Given a dataset of expert trajectories, standard imitation learning approaches typically learn a direct mapping from observations (e.g., RGB images) to actions. However, such methods often overlook the rich interplay between different modalities, i.e., sensory inputs, actions, and rewards, which is crucial for modeling robot behavior and understanding task outcomes. In this work, we propose Multimodal Diffusion Forcing, a unified framework for learning from multimodal robot trajectories that extends beyond action generation. Rather than modeling a fixed distribution, MDF applies random partial masking and trains a diffusion model to reconstruct the trajectory. This training objective encourages the model to learn temporal and cross-modal dependencies, such as predicting the effects of actions on force signals or inferring states from partial observations. We evaluate MDF on contact-rich, forceful manipulation tasks in simulated and real-world environments. Our results show that MDF not only delivers versatile functionalities, but also achieves strong performance, and robustness under noisy observations. More visualizations can be found on our website https://unified-df.github.io

Figures (7)

• Figure 1: We propose Multimodal Diffusion Forcing, a unified model that captures the interplay between modalities over time through masked diffusion training. At inference time, the model not only offers flexibility by allowing different input modalities, adjustable horizon lengths and prediction horizons, it also diverse functionalities—serving as a policy, planner, dynamics model, state estimator, and anomaly detector
• Figure 2: Regular diffusion models employ a scalar noise level to control the denoising process. Diffusion Forcing chen2024diffusion extends this idea with a time-varying noise vector to sample video sequences autoregressively. We further generalize this framework to the multimodal setting by introducing a time–modality varying noise matrix. This design enables versatile functionalities at test time such as policy, planner, dynamics model, and fine-grained anomaly detector.
• Figure 3: An overview of the key components and training process of MDF. Pretraining: MDF learns a diffusion-based autoencoder to compress point clouds into compact embeddings. Multimodal masked training: MDF processes six modalities: partial point cloud, full point cloud (training only), force, action, reward and proprioception (omitted in figure). The point clouds are tokenized using the pretrained PointNet encoder (frozen during MDF training). Data from all modalities are then concatenated and corrupted with noise according to a randomly sampled 2D noise level matrix. The diffusion transformer is trained to denoise this corrupted input, learning temporal and cross-modal dependencies.
• Figure 4: Contact-rich manipulaation tasks in IsaacSim.
• Figure 5: The history length of MDF can be adjusted dynamically at test time to accommodate task requirements.