Translating Flow to Policy via Hindsight Online Imitation

TL;DR

HinFlow tackles the data bottleneck in grounding high-level, video-derived flow plans to actionable robot policies by coupling online interaction with hindsight relabeling. A flow predictor trained on action-free videos supplies short-horizon, task-centric guidance, while a transformer-based flow-conditioned policy is iteratively refined through online rollouts and hindsight relabeling of achieved flows. Across LIBERO and ManiSkill3 benchmarks, HinFlow delivers strong, sample-efficient performance, including real-world validation and cross-embodiment transfer, demonstrating robust generalization to visual variations and unseen objects. This framework offers a scalable path to transferring video-based planning into robust robotic control without relying on extensive robot-centered demonstrations.

Abstract

Recent advances in hierarchical robot systems leverage a high-level planner to propose task plans and a low-level policy to generate robot actions. This design allows training the planner on action-free or even non-robot data sources (e.g., videos), providing transferable high-level guidance. Nevertheless, grounding these high-level plans into executable actions remains challenging, especially with the limited availability of high-quality robot data. To this end, we propose to improve the low-level policy through online interactions. Specifically, our approach collects online rollouts, retrospectively annotates the corresponding high-level goals from achieved outcomes, and aggregates these hindsight-relabeled experiences to update a goal-conditioned imitation policy. Our method, Hindsight Flow-conditioned Online Imitation (HinFlow), instantiates this idea with 2D point flows as the high-level planner. Across diverse manipulation tasks in both simulation and physical world, our method achieves more than $2\times$ performance improvement over the base policy, significantly outperforming the existing methods. Moreover, our framework enables policy acquisition from planners trained on cross-embodiment video data, demonstrating its potential for scalable and transferable robot learning.

Figures (24)

• Figure 1: Motivation of HinFlow. Learning from large-scale and diverse video data, a flow-based high-level planner can formulate generalizable task plans. To robustly execute these plans, the low-level control policy needs to be iteratively refined through online practice, preventing the overall system from being bottlenecked by low-level execution.
• Figure 2: Overview of HinFlow.Left: Hierarchical Policy. Our framework employs a flow prediction model to generate high-level plans in the form of point flow, which guide the low-level policy. Right: Hindsight Relabeled Replay Buffer. The robot rollouts the policy in the environment to collect explorative trajectories and retrospectively annotate the achieved flow subgoals using a video tracker. Subsequently, it performs policy updates conditioned on hindsight-relabeled flows, which in turn creates a virtuous cycle for Self-improvement.
• Figure 3: Visualization of all experiment tasks. The first four tasks are on LIBERO and the other three are on ManiSkill. The arrows indicate the the sequential steps required to complete each task.
• Figure 4: Performance comparison of HinFlow against baselines on the LIBERO and ManiSkill tasks. Online methods interact with the environment for 80000 steps. The shaded region represents the standard deviation across five random seeds. Notably, our method achieves significantly higher performance and sample efficiency.
• Figure 5: Visual setup of the real-world experiment. The robot is required to pick up a mouse and place it on a pad. We highlight the positions of the wrist-mounted and third-person cameras. The red bounding box indicates the $15\text{cm}\times 15\text{cm}$ region used for randomized object initialization during resets.