Latent Adaptive Planner for Dynamic Manipulation

TL;DR

The paper introduces Latent Adaptive Planner (LAP), a trajectory-level latent-variable policy for dynamic nonprehensile manipulation that learns from human demonstration videos. Planning is treated as latent-space inference, with a Transformer-based trajectory generator and classical variational Bayes learning, enabling efficient on-line variational replanning as new observations arrive. A robot-model-based data-regeneration pipeline converts human demonstrations into robot-ready kinematic and dynamic data, enabling cross-platform transfer without requiring robot-specific data. Experiments on box catching show LAP achieves high success, smooth trajectories, and energy-efficient behavior, outperforming BC and diffusion baselines while rivaling model-based planning in effectiveness. The approach promises scalable visuomotor learning with real-time adaptation and robust cross-robot transfer by combining data regeneration with latent planning.

Abstract

We present the Latent Adaptive Planner (LAP), a trajectory-level latent-variable policy for dynamic nonprehensile manipulation (e.g., box catching) that formulates planning as inference in a low-dimensional latent space and is learned effectively from human demonstration videos. During execution, LAP achieves real-time adaptation by maintaining a posterior over the latent plan and performing variational replanning as new observations arrive. To bridge the embodiment gap between humans and robots, we introduce a model-based proportional mapping that regenerates accurate kinematic-dynamic joint states and object positions from human demonstrations. Through challenging box catching experiments with varying object properties, LAP demonstrates superior success rates, trajectory smoothness, and energy efficiency by learning human-like compliant motions and adaptive behaviors. Overall, LAP enables dynamic manipulation with real-time adaptation and successfully transfer across heterogeneous robot platforms using the same human demonstration videos.

Figures (5)

• Figure 1: Robot Model-Based Data Regeneration Pipeline from Human Demonstration Videos. The pipeline consists of three main stages: (1) Scene State Estimation, which detects and tracks box objects and human pose from demonstration videos; (2) Object-Robot Proportional Mapping, which scales object dimensions and positions relative to the robot base frame; and (3) Kinematic-Dynamic Joint State Reconstruction, which maps human joint positions to robot configurations, differentiates to obtain velocities and accelerations, and computes required joint torques through inverse dynamics including external forces.
• Figure 2: System Architecture for LAP Framework. The diagram illustrates our perception-planning-control pipeline. Camera input undergoes segmentation for object detection, providing box states to the Latent Adaptive Planner (LAP). LAP operates on a dual-rate hierarchy, performing updating of latent plan at 30Hz while generating motion commands at 100Hz. Reference joint positions, velocities, and torques from LAP are refined by Model Predictive Control before execution at the motor level. Joint states and contact state are fed back to the LAP for replanning. This architecture ensures smooth execution of dynamic nonprehensile manipulation tasks while respecting robot dynamics and physical constraints, enabling real-time adaptation to environmental changes.
• Figure 3: Impact-aware retreat trajectory for box catching motion learned with the Latent Adaptive Planner (LAP). The left panel shows a human demonstration where the subject absorbs impact by yielding their arm along the trajectory of the incoming box before returning to the nominal pose. The right panel shows the robot reproducing this compliant motion using LAP. The red curves indicate the impact-aware retreat trajectory that minimizes energy consumption during dynamic interaction.
• Figure 4: Coordinate system of Robot A
• Figure 5: Coordinate system of Robot B