Temporally Coherent Imitation Learning via Latent Action Flow Matching for Robotic Manipulation

TL;DR

This work tackles the challenge of real-time, long-horizon robotic manipulation by introducing LG-Flow Policy, which performs flow matching in a continuous latent action space. By encoding action sequences into temporally regular latent trajectories and learning an explicit latent-space flow, the method achieves near single-step inference while improving trajectory smoothness and stability. It further enhances perception and execution with geometry-aware point-cloud conditioning and visual modulation during decoding, validated through extensive simulation and real-world experiments that show superior task success and competitive latency compared with diffusion-based and raw-action baselines. The approach offers a practical path toward stable, expressive, and efficient manipulation in dynamic environments, with potential extensions to richer multimodal cues and contact-aware latent dynamics.

Abstract

Learning long-horizon robotic manipulation requires jointly achieving expressive behavior modeling, real-time inference, and stable execution, which remains challenging for existing generative policies. Diffusion-based approaches provide strong modeling capacity but typically incur high inference latency, while flow matching enables fast one-step generation yet often leads to unstable execution when applied directly in the raw action space. We propose LG-Flow Policy, a trajectory-level imitation learning framework that performs flow matching in a continuous latent action space. By encoding action sequences into temporally regularized latent trajectories and learning an explicit latent-space flow, the proposed approach decouples global motion structure from low-level control noise, resulting in smooth and reliable long-horizon execution. LG-Flow Policy further incorporates geometry-aware point cloud conditioning and execution-time multimodal modulation, with visual cues evaluated as a representative modality in real-world settings. Experimental results in simulation and on physical robot platforms demonstrate that LG-Flow Policy achieves near single-step inference, substantially improves trajectory smoothness and task success over flow-based baselines operating in the raw action space, and remains significantly more efficient than diffusion-based policies.

Figures (8)

• Figure 1: Overall architecture of the proposed LG-Flow Policy. The system first encodes point cloud observations into geometry-aware scene features, then performs latent-space flow matching to generate temporally coherent latent action trajectories, which are finally decoded into executable control commands under visual conditioning.
• Figure 2: Conditional variational autoencoder for latent action modeling. A GRU encoder enforces temporal coherence in the latent space, while an MLP decoder with FiLM-conditioned wrist-camera features enables visually adaptive action execution.
• Figure 3: Geometry-aware point cloud encoder. Local neighborhoods around farthest-sampled centers capture translation-invariant local geometry via residual convolutions and max--mean pooling, while a lightweight center encoder provides compact global scene context for latent trajectory conditioning.
• Figure 4: Trajectory smoothness comparison across simulated manipulation tasks. Lower values indicate smoother execution.
• Figure 5: Real-world experimental setup and observations. Left: Franka Emika Panda robot with a LEAP Hand and the visual sensing setup (global L515 and wrist-mounted D435). Right: Example multimodal observations (RGB images, point clouds, wrist views) and task objects used in real-world experiments.