Flow Policy Gradients for Robot Control

TL;DR

Flow policy gradients provide a likelihood-free route to train expressive flow-based policies for robotic control. The authors introduce FPO++, an enhancement with per-sample ratio clipping and an asymmetric trust region to stabilize training across locomotion, humanoid sim-to-real, and manipulation fine-tuning, plus zero-sampling at test time to reduce latency. Experiments demonstrate robust sim-to-real transfer on humanoids and improved exploration and robustness over Gaussian PPO baselines, with ablations highlighting the importance of per-sample ratios and ASPO. This work challenges the necessity of explicit likelihoods in policy gradient RL for robotics, expanding the design space for flow-based reinforcement learning in real-world control systems.

Abstract

Likelihood-based policy gradient methods are the dominant approach for training robot control policies from rewards. These methods rely on differentiable action likelihoods, which constrain policy outputs to simple distributions like Gaussians. In this work, we show how flow matching policy gradients -- a recent framework that bypasses likelihood computation -- can be made effective for training and fine-tuning more expressive policies in challenging robot control settings. We introduce an improved objective that enables success in legged locomotion, humanoid motion tracking, and manipulation tasks, as well as robust sim-to-real transfer on two humanoid robots. We then present ablations and analysis on training dynamics. Results show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.

Figures (23)

• Figure 1: Flow policies for robot control. We show how robust control policies for quadrupeds, humanoids, and manipulators can be trained and deployed with the flow matching policy gradient framework mcallister2025flow.
• Figure 2: Improved stability in IsaacLab locomotion environments. We compare episode returns from FPO++ training with FPO returns over many different hyperparameter choices: FPO++ rewards are averaged over 5 seeds, while FPO runs are included for all combinations of learning rate $\in \{10^{-5}, 10^{-4}, 3\times 10^{-4}\}$, clip parameter $\in \{0.04,0.05,0.06\}$, and Monte Carlo samples $\in \{8, 16, 32\}$. FPO++ addresses stability problems that we were unable to solve by tuning FPO hyperparameters.
• Figure 3: Sim-to-real transfer. We deploy flow policies for locomotion to a Booster T1 and motion tracking to a Unitree G1. Policies are directly deployed to real robots with reduced sampling step counts, demonstrating stable gaits, tracking for long sequences, and robustness to external forces. The arrows in T1 locomotion (\ref{['fig:t1_locomotion']}) indicate velocity commands, while the arrows in the robustness tests (\ref{['fig:G1_robust_test']}) highlight external forces.
• Figure 4: Manipulation fine-tuning results. We compare the evaluation success rates for FPO++, FPO, and our two DPPO implementations. We show two rows: the first row contains policy success rates with zero-sampling ($\epsilon = \vec{0}$), while the second row contains policy success rates using standard random sampling ($\epsilon \sim \mathcal{N}(0,I)$). For each task, all algorithms are initialized from the same base policy, which is an image-based flow matching policy trained to predict action chunks.
• Figure 5: Per-sample ratios improve locomotion policies. We plot final training and evaluation returns, comparing FPO++ with per-sample ratios (Equation \ref{['eq:per_sample_ratio']}) against per-action ratios from prior work (Equation \ref{['eq:per_action_ratio']}). We show results for many training runs: each point is a training run with clipping parameter sampled $\in \{0.04, 0.05, 0.06\}$ and random seed $\in \{0,1,2,3,4\}$. Per-sample ratios produce higher and more consistent returns across environments.