Dynamic Manipulation of Deformable Objects in 3D: Simulation, Benchmark and Learning Strategy

TL;DR

This work proposes Dynamics Informed Diffusion Policy (DIDP), a framework that integrates imitation pretraining with physics-informed test-time adaptation and designs a diffusion policy that learns inverse dynamics within the reduced-order space, enabling imitation learning to move beyond na\"ive data fitting and capture the underlying physical structure.

Abstract

Goal-conditioned dynamic manipulation is inherently challenging due to complex system dynamics and stringent task constraints, particularly in deformable object scenarios characterized by high degrees of freedom and underactuation. Prior methods often simplify the problem to low-speed or 2D settings, limiting their applicability to real-world 3D tasks. In this work, we explore 3D goal-conditioned rope manipulation as a representative challenge. To mitigate data scarcity, we introduce a novel simulation framework and benchmark grounded in reduced-order dynamics, which enables compact state representation and facilitates efficient policy learning. Building on this, we propose Dynamics Informed Diffusion Policy (DIDP), a framework that integrates imitation pretraining with physics-informed test-time adaptation. First, we design a diffusion policy that learns inverse dynamics within the reduced-order space, enabling imitation learning to move beyond naïve data fitting and capture the underlying physical structure. Second, we propose a physics-informed test-time adaptation scheme that imposes kinematic boundary conditions and structured dynamics priors on the diffusion process, ensuring consistency and reliability in manipulation execution. Extensive experiments validate the proposed approach, demonstrating strong performance in terms of accuracy and robustness in the learned policy.

Figures (5)

• Figure 1: We introduce the 3D dynamic manipulation benchmark for deformable objects by leveraging GVS modeling. This framework enables a reduced-DoF representation and differentiable system modeling, facilitating efficient learning of inverse dynamics. Our approach ensures consistency between kinematics and dynamics throughout the policy learning process.
• Figure 2: Pipeline of the Proposed DIDP Framework. $D$ denotes the DoF of the entire system, and $N$ the length of the action sequence. In the denoising network $\epsilon_\eta$, the red blocks represent learnable parameters, while blue blocks indicate frozen modules. During the training phase, all parameters are optimized jointly. During the inference phase, we employ a test-time adaptation strategy by fine-tuning only the final projection layer.
• Figure 3: Dataset overview. (a) shows the distribution of goal conditions in 3D space for both training and testing cases, demonstrating comprehensive spatial coverage. (b) and (c) depict the distributions of kinematic and dynamic states, respectively. The curves indicate the mean and standard deviation across time steps, based on a reduced representation with 20 DoFs to describe the entire system. (d) provides a visualization of the complete system within the simulation environment.
• Figure 4: A case study comparing learning strategies. TO improves IL's action accuracy but fails to produce correct actions on its own.
• Figure 5: Visualization of the manipulation process. The baseline represents the diffusion policy without test-time adaptation. DDP denotes test-time adaptation using only the differentiable dynamics prior, while KBC indicates adaptation incorporating the kinematic boundary condition.

Theorems & Definitions (1)

• Corollary 1