ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulation

TL;DR

Diffusion policies for robotic manipulation operate on the $SE(3)$ manifold, but offline training and unconstrained denoising limit generalization and efficiency. We present ADP, a training-free test-time adaptive diffusion policy that adds (i) task-manifold guided denoising along geodesics, (ii) a Gaussian spherical prior to constrain step sizes, and (iii) a task-aware initialization via Fast Global Registration between test-time gripper and scene point clouds. Implemented as ADPro by plugging into a pretrained 3D Diffuser Actor (and optionally CLIP-based prompts), ADP yields up to ~25% faster inference and ~9 percentage-point improvements on RLBench, CALVIN, and RealWP, with strong zero-shot generalization. These results show that geometry-aware, on-the-fly guidance can substantially enhance manipulation performance without retraining, widening the practical applicability of diffusion policies in unstructured environments.

Abstract

Diffusion policies have recently emerged as a powerful class of visuomotor controllers for robot manipulation, offering stable training and expressive multi-modal action modeling. However, existing approaches typically treat action generation as an unconstrained denoising process, ignoring valuable a priori knowledge about geometry and control structure. In this work, we propose the Adaptive Diffusion Policy (ADP), a test-time adaptation method that introduces two key inductive biases into the diffusion. First, we embed a geometric manifold constraint that aligns denoising updates with task-relevant subspaces, leveraging the fact that the relative pose between the end-effector and target scene provides a natural gradient direction, and guiding denoising along the geodesic path of the manipulation manifold. Then, to reduce unnecessary exploration and accelerate convergence, we propose an analytically guided initialization: rather than sampling from an uninformative prior, we compute a rough registration between the gripper and target scenes to propose a structured initial noisy action. ADP is compatible with pre-trained diffusion policies and requires no retraining, enabling test-time adaptation that tailors the policy to specific tasks, thereby enhancing generalization across novel tasks and environments. Experiments on RLBench, CALVIN, and real-world datasets show that ADPro, an implementation of ADP, improves success rates, generalization, and sampling efficiency, achieving up to 25% faster execution and 9% points over strong diffusion baselines.

Figures (6)

• Figure 1: Comparison of (a) vanilla DPs and (b) our ADP. Vanilla DPs generate robot actions by progressively denoising from a random initialization, but this process typically unfolds in an unconstrained Euclidean space. In contrast, our ADP employs task-aware initialization and constrains updates along both task and spherical manifolds, yielding a more generalizable policy.
• Figure 2: The pipeline of our adaptive diffusion policy ADPro. We first propose reasonable initial noisy actions $\mathbf{a}_{M}$ with the rough registration module, then, denoise $\mathbf{a}_{M}$ under the guidance of observations and manifold constraint (see Eq. \ref{['eq:fdpmin']} for details).
• Figure 3: Trajectory comparison between the vanilla diffusion policy and our ADPro. We illustrate two distinct tasks: 'sort shape' and 'insert peg'. For each task, we visualize the initial state followed by successive observations captured from the wrist-mounted camera after each action. ADPro completes the first task with only three actions, demonstrating both efficiency and accuracy.
• Figure 4: Comparison on all components ($x$, $y$, $z$, $r_1$, $r_2$, $r_3$, $r_4$, $w$) of the full action for tasks 'open drawer' and 'sweep to dustpan'. The horizontal and vertical axes represent action steps and parameter values, respectively. Our ADPro effectively mitigates coordinate and angle backtracking behaviors in the vanilla diffusion policy.
• Figure 5: Diffusion-step evolution for the first four actions in task 'sweep to dustpan'. The MSE of Diffuser exhibits large fluctuations, indicating significant backtracking behavior in its trajectory.