F2F-AP: Flow-to-Future Asynchronous Policy for Real-time Dynamic Manipulation

Abstract

Asynchronous inference has emerged as a prevalent paradigm in robotic manipulation, achieving significant progress in ensuring trajectory smoothness and efficiency. However, a systemic challenge remains unresolved, as inherent latency causes generated actions to inevitably lag behind the real-time environment. This issue is particularly exacerbated in dynamic scenarios, where such temporal misalignment severely compromises the policy's ability to interpret and react to rapidly evolving surroundings. In this paper, we propose a novel framework that leverages predicted object flow to synthesize future observations, incorporating a flow-based contrastive learning objective to align the visual feature representations of predicted observations with ground-truth future states. Empowered by this anticipated visual context, our asynchronous policy gains the capacity for proactive planning and motion, enabling it to explicitly compensate for latency and robustly execute manipulation tasks involving actively moving objects. Experimental results demonstrate that our approach significantly enhances responsiveness and success rates in complex dynamic manipulation tasks.

Figures (12)

• Figure 1: Flow-to-Future Asynchronous Policy (F2F-AP). F2F-AP is an asynchronous robot manipulation policy that explicitly incorporates future observations. By predicting future observations in the form of optical flow, it enhances the model's understanding of the motion trends of interacting objects. F2F-AP is transferable to diverse embodiments and yields significant performance improvements in dynamic tasks.
• Figure 2: Decomposition of system latency. The total latency ($\Delta_o + \Delta_i + \Delta_c$) results in a temporal misalignment where the initial frames of the predicted trajectory lag behind the real-world state, invalidating them for execution.
• Figure 3: Analysis of Asynchronous Inference. The policy $\pi(\mathbf{a} | \mathbf{s}, \mathbf{o})$ maps inputs of observations and robot states to action sequences. From left to right, the policy sequentially incorporates future states and predicted observations as augmented context. These additions respectively resolve the trajectory discontinuity caused by state lag and the temporal misalignment of actions resulting from information lag.
• Figure 4: Overview of the pipeline.Left: Illustration of the asynchronous inference achieved by F2F-AP. The model plans from a future state $s_{t_3}$ towards the anticipated position $t_6$ of the interacting object at timestamp $t_1$, enabling advance planning and motion despite real-world system latency. Middle: The model takes robot states and multi-frame RGB images as input. A Flow Predictor extracts object flow to synthesize augmented future observations, which are then processed by the Policy as future observation to generate action chunks. Right:We introduce contrastive learning to minimize the feature distance between predicted and real future observations. the $\bigstar$ indicates that these features share the same encoder.
• Figure 5: Hardware Platform. F2F-AP is evaluated in fixed-base arm and quadruped manipulator equipped with camera and odometry.