M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes

TL;DR

M2Diffuser introduces a diffusion-based, scene-conditioned planner for mobile manipulation in 3D scenes, generating whole-body trajectories directly from robot-centric 3D scans. By integrating differentiable energy and cost functions into guided diffusion sampling, it enforces physical constraints (collision avoidance, joint limits, smoothness) and task objectives (grasping, placement, goal-reaching) during inference. Trained on expert planner trajectories and evaluated in both simulated and real environments, M2Diffuser outperforms autoregressive neural planners and demonstrates sim-to-real transfer without fine-tuning, while offering flexibility to new tasks. The work highlights the potential of diffusion priors for high-dimensional robotic planning and emphasizes the importance of explicit constraint enforcement for safe, robust execution in complex 3D environments.

Abstract

Recent advances in diffusion models have opened new avenues for research into embodied AI agents and robotics. Despite significant achievements in complex robotic locomotion and skills, mobile manipulation-a capability that requires the coordination of navigation and manipulation-remains a challenge for generative AI techniques. This is primarily due to the high-dimensional action space, extended motion trajectories, and interactions with the surrounding environment. In this paper, we introduce M2Diffuser, a diffusion-based, scene-conditioned generative model that directly generates coordinated and efficient whole-body motion trajectories for mobile manipulation based on robot-centric 3D scans. M2Diffuser first learns trajectory-level distributions from mobile manipulation trajectories provided by an expert planner. Crucially, it incorporates an optimization module that can flexibly accommodate physical constraints and task objectives, modeled as cost and energy functions, during the inference process. This enables the reduction of physical violations and execution errors at each denoising step in a fully differentiable manner. Through benchmarking on three types of mobile manipulation tasks across over 20 scenes, we demonstrate that M2Diffuser outperforms state-of-the-art neural planners and successfully transfers the generated trajectories to a real-world robot. Our evaluations underscore the potential of generative AI to enhance the generalization of traditional planning and learning-based robotic methods, while also highlighting the critical role of enforcing physical constraints for safe and robust execution.

Figures (11)

• Figure 1: Overview of the M$^2$Diffuser, a diffusion-based motion planner designed to sample and optimize whole-body coordinated trajectories directly from natural 3D scans, efficacious for mobile manipulation in 3D scenes. Using robot-centric 3D scans as visual input, M$^2$Diffuser employs an iterative denoising process to generate task-specific trajectories. It optimizes the sampled results at each denoising diffusion step guided by cost and energy functions, ensuring physical plausibility and task completion of generated trajectories.
• Figure 2: The diffusion and denoising process of M$^2$Diffuser. The example shows the diffusion and denoising process of the robot's end effector trajectory in a grasping task (e.g., grasping a book).
• Figure 3: Dataset collection procedure. (a) The Task Builder enables the construction of mobile manipulation tasks through high-level configurations, including scene and robot URDF, manipulated object link, target end effector goal, and task type. (b) The Expert Solver computes optimal whole-body coordinated trajectories by leveraging the vkc algorithm jiao2021consolidatingjiao2021efficient. (c) The Data Collector is responsible for recording the planned trajectories, and processing the segmented point clouds cropped from the perfect 3D scan based on the bounding box around the robot's initial position.
• Figure 4: Quantitative evaluation metrics of grasping and placement tasks. Previous work batra2020rearrangementehsani2021manipulathorhuang2023skillni2023towards evaluate object grasping by the contact between the end effector's bounding sphere and the object surface. This evaluation strategy often fails to reflect how the grasping performs in real-world scenarios. In this paper, we evaluate object grasping and placement quality by success rate in simulated scenes with physical simulation enabled. We use NVIDIA Isaac Sim as the physical simulator.
• Figure 5: Successful trajectories generated by M$^2$Diffuser on object grasping and placement tasks. These figures illustrate the successful trajectories generated by our method in (a) grasping and (b) placement tasks involving various objects.