Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion

TL;DR

This work tackles the challenge of generating collision-free, dynamically feasible motions in cluttered environments under tight time constraints. It introduces a two-stage diffusion-guided MPC framework that first uses a scene-conditioned diffusion transformer to produce a warm-start trajectory and then refines it with a collision-aware receding-horizon MPC, enforcing $\bm{x}_{t+1}=f_t(\bm{x}_t,\bm{u}_t)$ and hard constraints $c_{t,i,j}(\bm{x}_t)\ge0$. A key contribution is the object-centric conditioning via Slot Attention, which yields compact latent obstacle representations to guide diffusion and improve generalization. The method is validated on a Franka Emika Panda in cluttered environments, achieving higher success rates and lower latency than baselines, and demonstrated in real hardware with reliable, safe execution. This combination of generative priors and model-based control enables real-time, collision-aware manipulation and offers a scalable path toward robust planning in complex, dynamic scenes.

Abstract

Acting in cluttered environments requires predicting and avoiding collisions while still achieving precise control. Conventional optimization-based controllers can enforce physical constraints, but they struggle to produce feasible solutions quickly when many obstacles are present. Diffusion models can generate diverse trajectories around obstacles, yet prior approaches lacked a general and efficient way to condition them on scene structure. In this paper, we show that combining diffusion-based warm-starting conditioned with a latent object-centric representation of the scene and with a collision-aware model predictive controller (MPC) yields reliable and efficient motion generation under strict time limits. Our approach conditions a diffusion transformer on the system state, task, and surroundings, using an object-centric slot attention mechanism to provide a compact obstacle representation suitable for control. The sampled trajectories are refined by an optimal control problem that enforces rigid-body dynamics and signed-distance collision constraints, producing feasible motions in real time. On benchmark tasks, this hybrid method achieved markedly higher success rates and lower latency than sampling-based planners or either component alone. Real-robot experiments with a torque-controlled Panda confirm reliable and safe execution with MPC.

Figures (8)

• Figure 1: Collision-free MPC with diffusion-generated warm-start. The robot, sequentially reaches multiple target end-effector positions (colored dots) using collision-free MPC. The learned priors allow effective control within the non-convex space of the cluttered shelf environment.
• Figure 2: Overview of the proposed architecture for robot control. Our objective is to control a robot (D) using an MPC feedback loop that solves an optimal control problem (C) over the robot's dynamics, denoted by $f_t(\cdot)$, while strictly enforcing obstacle-avoidance constraints $c_t(\cdot,\cdot)$. Effective warm-starting is crucial for enabling the MPC solver to converge to a feasible, collision-free solution. To this end, we employ a diffusion model (B) that denoises an initial noise sequence $\bm{\hat{Q}}_K$ to generate a warm-start trajectory $\bm{\hat{Q}^\star}$, thereby accelerating convergence. To incorporate environmental context, we propose an object-centric conditioning mechanism (A) based on Slot Attention. This conditioning takes as input the target end-effector position $\bm{p_\text{goal}}$, the current robot state $\bm{x}_0$, and a latent representation of the environment derived from the estimated object poses $\bm T^i_{WO}$ via Slot Attention. The Slot Attention model internally renders a synthetic image from these poses, which helps mitigate the real-to-sim gap between simulated training data and real-world deployment.
• Figure 3: Object-centric scene representation using Slot Attention. Given an RGB image of the scene, a pre-trained Slot-Attention encoder locatello_object-centric_2020 extracts a set of object-centric latent embeddings, called slots, each corresponding to a distinct object or obstacle in the environment. These slots are then used to construct a structured representation of the scene, which serves as input to the diffusion-based trajectory generator. This conditioning enables the model to generate trajectories that account for obstacle geometry and layout, while maintaining generalization to unseen environments.
• Figure 4: State-of-the-art methods comparison. We use Table, Drawer, and Shelf benchmarks shown on the left to compare our diffusion-warm-started MPC against PRESTO seo_presto_2025, Motion-Planning Diffusion (MPD) carvalho_motion_2023, cuRobo sundaralingam_curobo_2023, RRT-Connect rrt-connect_kuffner_2000 and a DDP-based OCP without a warm-start mastalli_crocoddyl_2020. Metrics are aggregated over 3 different scenes with various obstacles positions. Higher is better for success rate (↑), lower is better (↓) for average cost and computation time. Our method attains the highest success on all levels, 82%, 79%, 83%, with the lowest costs and sub-72ms runtime. PRESTO seo_presto_2025, cuRobo sundaralingam_curobo_2023 and RRT-Connect rrt-connect_kuffner_2000 are slower but still provide high success rates over 70%, while MPD and the pure OCP baseline remain below 10% success.
• Figure 5: Ablation of scene representations. We evaluated our control architecture using different scene representations. The Slot Attention-based representation achieved the best performance, especially in low-computation scenarios. The configuration-space representation performed nearly as well, while image-based conditioning was uncompetitive for control applications.