PocketDP3: Efficient Pocket-Scale 3D Visuomotor Policy

TL;DR

PocketDP3 tackles parameter inefficiency in 3D diffusion visuomotor policies by replacing the heavy conditional U-Net decoder with a lightweight Diffusion Mixer (DiM) built on MLP-Mixer blocks, while preserving a compact scene encoder. It demonstrates two-step inference without distillation via DDIM sampling, achieving strong results across RoboTwin2.0, Adroit, and MetaWorld with a parameter count of less than $0.01$ of prior methods. Real-world experiments on an AgileX Piper show promising sim-to-real transfer, validating practical deployment. This work indicates that decoder efficiency and robust fusion of temporal and channel information can make diffusion-based visuomotor control viable on resource-constrained robots.

Abstract

Recently, 3D vision-based diffusion policies have shown strong capability in learning complex robotic manipulation skills. However, a common architectural mismatch exists in these models: a tiny yet efficient point-cloud encoder is often paired with a massive decoder. Given a compact scene representation, we argue that this may lead to substantial parameter waste in the decoder. Motivated by this observation, we propose PocketDP3, a pocket-scale 3D diffusion policy that replaces the heavy conditional U-Net decoder used in prior methods with a lightweight Diffusion Mixer (DiM) built on MLP-Mixer blocks. This architecture enables efficient fusion across temporal and channel dimensions, significantly reducing model size. Notably, without any additional consistency distillation techniques, our method supports two-step inference without sacrificing performance, improving practicality for real-time deployment. Across three simulation benchmarks--RoboTwin2.0, Adroit, and MetaWorld--PocketDP3 achieves state-of-the-art performance with fewer than 1% of the parameters of prior methods, while also accelerating inference. Real-world experiments further demonstrate the practicality and transferability of our method in real-world settings. Code will be released.

Figures (5)

• Figure 1: Comparison of PocketDP3 with the state-of-the-art 2D-based method DPchi2025diffusion and 3D-based methods DP3ze20243d, FlowPolicyzhang2025flowpolicy in terms of inference latency/model size and average success rate on RoboTwin2.0, Adroit, and MetaWorld.
• Figure 2: Overall architecture of PocketDP3. In the figure, $\rm T$ denotes transpose.Our PocketDP3 adopts the efficient point-cloud encoder from DP3 ze20243d and stacks $K$ DiM blocks as the decoder. Each DiM block is built upon an MLP-Mixer style tolstikhin2021mlp architecture, enabling efficient information fusion with a small parameter budget, thereby improving decision-making performance.
• Figure 3: Visualizations of the simulation experiments. The simulation results show that our PocketDP3 achieves strong performance across a variety of tasks, demonstrating the efficiency of its architecture design.
• Figure 4: Real-World Experiment Setup.
• Figure 5: Real-world Experiments. The image sequence (top to bottom) illustrates the robot successfully performing three tasks: placing an object, uprighting a fallen cup, and stacking two blocks.