Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning

Shan Zhong, Shutong Ding, He Diao, Xiangyu Wang, Kah Chan Teh, Bei Peng

TL;DR

Inspired by flow matching's success in generative modeling, this work proposes a generative paradigm for value estimation, named FlowCritic, which leverages flow matching to model value distributions and generate samples for value estimation.

Abstract

Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, FlowCritic leverages flow matching to model value distributions and generate samples for value estimation.

FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning

TL;DR

Inspired by flow matching's success in generative modeling, this work proposes a generative paradigm for value estimation, named FlowCritic, which leverages flow matching to model value distributions and generate samples for value estimation.

Abstract

Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, FlowCritic leverages flow matching to model value distributions and generate samples for value estimation.
Paper Structure (18 sections, 6 theorems, 75 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 18 sections, 6 theorems, 75 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Online Reinforcement Learning
  4. Flow Matching Methods
  5. Methods
  6. Value Distribution Modeling via Flow Matching
  7. Robust Value Learning with Truncation and Clipping
  8. Weighted Policy Optimization via Value Distribution Modeling
  9. Theoretical Analysis
  10. Experiments
  11. Value Estimation in a Single-Step Environment
  12. Experimental Setup for Continuous Control Tasks
  13. Performance Comparisons
  14. Ablation Studies
  15. Core Component Ablation
  16. ...and 3 more sections

Key Result

Lemma 1

If $X' = X+c$ and $Y' = Y+c$ with corresponding distributions $\mu'$ and $\nu'$, then

Figures (6)

  • Figure 1: FlowCritic weighting mechanism. Value distributions with lower noise levels receive larger weights in policy optimization, ensuring that reliable estimates dominate the learning process.
  • Figure 2: Results on the single-step environment.
  • Figure 3: Learning curves comparisons of FlowCritic (purple), PPO (red), PPO_AVC (blue), PPO_CVE (green), and PPO_QD (yellow) on 12 control benchmark tasks. Solid lines represent mean episodic returns over 5 random seeds, with shaded areas indicating one standard deviation.
  • Figure 4: Ablation study on key components of FlowCritic.
  • Figure 5: Sensitivity analysis of key hyperparameters of FlowCritic.
  • ...and 1 more figures

Theorems & Definitions (11)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Theorem 1: Convergence of Approximate Value Distribution Iteration
  • proof
  • Corollary 1
  • proof
  • Theorem 2: Variance Reduction via CoV Weighting
  • ...and 1 more