ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervision

TL;DR

This work tackles efficiency and generalization gaps in vision-based imitation by leveraging a 3D diffusion policy that operates on sparse point clouds. It introduces Implicit Scene Supervision (ISS), a skip-step future-scene predictor that guides the DiT-based policy toward geometrically consistent actions, improving long-horizon reasoning. Across MetaWorld and Adroit, ISS Policy achieves state-of-the-art results, with strong real-world performance and rapid training convergence, while ablations confirm benefits from ISS, scheduling, and increased model capacity. The approach demonstrates scalable, robust 3D visuomotor control from single-view depth data and offers practical speed advantages for real-time robotics tasks such as cup-stacking.

Abstract

Vision-based imitation learning has enabled impressive robotic manipulation skills, but its reliance on object appearance while ignoring the underlying 3D scene structure leads to low training efficiency and poor generalization. To address these challenges, we introduce \emph{Implicit Scene Supervision (ISS) Policy}, a 3D visuomotor DiT-based diffusion policy that predicts sequences of continuous actions from point cloud observations. We extend DiT with a novel implicit scene supervision module that encourages the model to produce outputs consistent with the scene's geometric evolution, thereby improving the performance and robustness of the policy. Notably, ISS Policy achieves state-of-the-art performance on both single-arm manipulation tasks (MetaWorld) and dexterous hand manipulation (Adroit). In real-world experiments, it also demonstrates strong generalization and robustness. Additional ablation studies show that our method scales effectively with both data and parameters. Code and videos will be released.

Figures (10)

• Figure 1: ISS Policy is a novel 3D visuomotor diffusion-based policy that generates continuous action sequences from point cloud inputs. It exhibits strong scalability and learning efficiency, achieving state-of-the-art performance on challenging simulation benchmarks and demonstrating strong robustness in real-world experiments.
• Figure 2: Model Architecture of ISS Policy. Expert demonstrations of point clouds and robot states are first encoded into observation context. During training, a diffusion noise scheduler adds noise to expert action trajectories, and a DiT-based policy head conditions on the context and diffusion timestep $t$ to denoise them and predict future action sequences. After the DiT policy produces a candidate trajectory, an implicit scene supervision (ISS) head takes the global point-cloud context together with a length-$K$ subsequence of predicted actions and predicts a skip-step future point-cloud embedding, providing an auxiliary future prediction signal that shapes the learned representations during training.
• Figure 3: Policy Formulation.
• Figure 4: Visualization of 3D point-cloud observations in simulation environments. We show representative 3D point clouds from Adroit and MetaWorld.
• Figure 5: Learning Efficiency and Stability. We plot training curves of ISS Policy and DP3 on MetaWorld tasks. The dashed horizontal lines denote the SR$_5$ metric. Across all tasks, ISS Policy achieves higher success rates, higher learning efficiency, and better asymptotic performance.