Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation

Guojian Zhan, Letian Tao, Pengcheng Wang, Yixiao Wang, Yiheng Li, Yuxin Chen, Masayoshi Tomizuka, Shengbo Eben Li

TL;DR

This work proposes mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation and theoretically proves that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness.

Abstract

Learning expressive and efficient policy functions is a promising direction in reinforcement learning (RL). While flow-based policies have recently proven effective in modeling complex action distributions with a fast deterministic sampling process, they still face a trade-off between expressiveness and computational burden, which is typically controlled by the number of flow steps. In this work, we propose mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation. To ensure its high expressiveness, an instantaneous velocity constraint (IVC) is introduced on the mean velocity field during training. We theoretically prove that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness. Empirically, our MVP achieves state-of-the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench. It also delivers substantial improvements in training and inference speed over existing flow-based policy baselines.

Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation

TL;DR

This work proposes mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation and theoretically proves that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness.

Abstract

Learning expressive and efficient policy functions is a promising direction in reinforcement learning (RL). While flow-based policies have recently proven effective in modeling complex action distributions with a fast deterministic sampling process, they still face a trade-off between expressiveness and computational burden, which is typically controlled by the number of flow steps. In this work, we propose mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation. To ensure its high expressiveness, an instantaneous velocity constraint (IVC) is introduced on the mean velocity field during training. We theoretically prove that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness. Empirically, our MVP achieves state-of-the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench. It also delivers substantial improvements in training and inference speed over existing flow-based policy baselines.
Paper Structure (38 sections, 4 theorems, 45 equations, 8 figures, 6 tables, 1 algorithm)

This paper contains 38 sections, 4 theorems, 45 equations, 8 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement Learning.
  4. Flow Matching.
  5. Method
  6. mean velocity policy
  7. (1) How to imitate a given target action.
  8. The Instantaneous Velocity Constraint as a Boundary Condition
  9. Mean Flow Reinforcement Learning
  10. Experiments
  11. Main Experiment
  12. Benchmark.
  13. Baselines and our method.
  14. Main results.
  15. Ablation Study and Supplementary Experiments
  16. ...and 23 more sections

Key Result

Theorem 1

Let the new policy $\pi_{\text{new}}$ be derived from an old policy $\pi_{\text{old}}$ via the $N$-candidate generative update process described above. Under assumptions of a bounded Q-function error ($\epsilon_Q$), $L_Q$-Lipschitz continuity of the Q-function, and a bounded mean flow matching error where $V(s)$ is the state-value function, and $\Delta_N^{\pi_{\text{old}}}(s)$ is the best-of-$N$ a

Figures (8)

  • Figure 1: Performance and efficiency comparison on 9 robotic manipulation tasks.
  • Figure 2: Velocity field: blue arrows denote the mean velocity over a time interval, with red arrows representing the instantaneous velocity at a time point.
  • Figure 3: Training curves on benchmarks. The solid lines correspond to mean and shaded regions correspond to 95% confidence interval over five runs. The shadow background indicates the offline training phase, while the white background indicates the online training phase.
  • Figure 4: Training curves of ablation on the IVC.
  • Figure 5: Training curves of comparison with one-step flow.
  • ...and 3 more figures

Theorems & Definitions (10)

  • Theorem 1: mean velocity policy Improvement
  • proof
  • Theorem 2: Multiplicity of Solutions for the Mean Flow Identity
  • proof
  • Theorem 3: Uniqueness via the Instantaneous Velocity Constraint
  • proof
  • Lemma 1: Performance Difference Lemma kakade2002approximately
  • proof
  • proof
  • proof