Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matching

Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Xiao Ma

TL;DR

This work tackles the challenge of applying maximum-entropy reinforcement learning to iterative generative policies whose action densities are not directly accessible. It introduces Field Least-Energy Actor-Critic (FLAC), a likelihood-free framework that regularizes policy stochasticity through the kinetic energy of a velocity field and recasts policy optimization as a Generalized Schrödinger Bridge (GSB) problem relative to a high-entropy reference. By showing that minimizing path-space energy bounds the deviation of the induced terminal action distribution from the reference, FLAC derives an energy-regularized policy objective and an off-policy algorithm with automatic energy tuning via a Lagrangian multiplier. Empirically, FLAC achieves competitive or superior performance on high-dimensional benchmarks (e.g., DMControl, HumanoidBench) without requiring explicit density estimation, highlighting its practical impact for scalable, entropy-aware control with implicit policies.

Abstract

Iterative generative policies, such as diffusion models and flow matching, offer superior expressivity for continuous control but complicate Maximum Entropy Reinforcement Learning because their action log-densities are not directly accessible. To address this, we propose Field Least-Energy Actor-Critic (FLAC), a likelihood-free framework that regulates policy stochasticity by penalizing the kinetic energy of the velocity field. Our key insight is to formulate policy optimization as a Generalized Schrödinger Bridge (GSB) problem relative to a high-entropy reference process (e.g., uniform). Under this view, the maximum-entropy principle emerges naturally as staying close to a high-entropy reference while optimizing return, without requiring explicit action densities. In this framework, kinetic energy serves as a physically grounded proxy for divergence from the reference: minimizing path-space energy bounds the deviation of the induced terminal action distribution. Building on this view, we derive an energy-regularized policy iteration scheme and a practical off-policy algorithm that automatically tunes the kinetic energy via a Lagrangian dual mechanism. Empirically, FLAC achieves superior or comparable performance on high-dimensional benchmarks relative to strong baselines, while avoiding explicit density estimation.

FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matching

TL;DR

This work tackles the challenge of applying maximum-entropy reinforcement learning to iterative generative policies whose action densities are not directly accessible. It introduces Field Least-Energy Actor-Critic (FLAC), a likelihood-free framework that regularizes policy stochasticity through the kinetic energy of a velocity field and recasts policy optimization as a Generalized Schrödinger Bridge (GSB) problem relative to a high-entropy reference. By showing that minimizing path-space energy bounds the deviation of the induced terminal action distribution from the reference, FLAC derives an energy-regularized policy objective and an off-policy algorithm with automatic energy tuning via a Lagrangian multiplier. Empirically, FLAC achieves competitive or superior performance on high-dimensional benchmarks (e.g., DMControl, HumanoidBench) without requiring explicit density estimation, highlighting its practical impact for scalable, entropy-aware control with implicit policies.

Abstract

Iterative generative policies, such as diffusion models and flow matching, offer superior expressivity for continuous control but complicate Maximum Entropy Reinforcement Learning because their action log-densities are not directly accessible. To address this, we propose Field Least-Energy Actor-Critic (FLAC), a likelihood-free framework that regulates policy stochasticity by penalizing the kinetic energy of the velocity field. Our key insight is to formulate policy optimization as a Generalized Schrödinger Bridge (GSB) problem relative to a high-entropy reference process (e.g., uniform). Under this view, the maximum-entropy principle emerges naturally as staying close to a high-entropy reference while optimizing return, without requiring explicit action densities. In this framework, kinetic energy serves as a physically grounded proxy for divergence from the reference: minimizing path-space energy bounds the deviation of the induced terminal action distribution. Building on this view, we derive an energy-regularized policy iteration scheme and a practical off-policy algorithm that automatically tunes the kinetic energy via a Lagrangian dual mechanism. Empirically, FLAC achieves superior or comparable performance on high-dimensional benchmarks relative to strong baselines, while avoiding explicit density estimation.
Paper Structure (60 sections, 3 theorems, 54 equations, 8 figures, 3 tables)

This paper contains 60 sections, 3 theorems, 54 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Entropy-Regularized RL
  5. Iterative Generative Policies
  6. The Schr$\ddot{\text{o}}$dinger Bridge Problem
  7. Kinetic Energy and Path Constraint
  8. Stochastic Regime ($\sigma > 0$).
  9. Deterministic Regime ($\sigma \to 0$).
  10. Reinforcement Learning as a Generalized Schr$\ddot{\text{o}}$dinger Bridge Problem
  11. The Generalized Schr$\ddot{\text{o}}$dinger Bridge Formulation
  12. Energy-Regularized Policy Optimization
  13. Deriving the FLAC Objective.
  14. Field Least-Energy Actor-Critic
  15. Energy-Regularized Policy Iteration
  16. ...and 45 more sections

Key Result

Proposition 1

The optimal path measure $\mathbb{P}^*$ that minimizes Eq. eq:gsb_variational induces a terminal marginal distribution $p^*(X_1)$ of the form: where $\mu_1^{\mathrm{ref}}(X_1)$ is the marginal distribution of the reference process at $\tau=1$.

Figures (8)

  • Figure 1: Kinetic Energy Regularization Encourage Exploration. Toy example on a 2D multi-goal landscape. (Top) Unconstrained: The high-velocity field overpowers the intrinsic noise, forcing the policy to collapse into a single deterministic mode. (Bottom) FLAC: By penalizing kinetic energy, the policy is constrained to preserve stochasticity. This low-energy field successfully recovers the full multimodal distribution.
  • Figure 2: Main results. We provide performance comparisons on two challenging benchmarks. For comprehensive results, please refer to Appendix D. All model-free algorithms are evaluated with 5 random seeds, while the model-based algorithm (TD-MPC2) uses 3 seeds. DIME incorporates cross Q-learning simmons2019q to boost performance, whereas FLAC does not rely on these enhancements.
  • Figure 3: Ablation Studies. (a) Sensitivity to target energy budget $E_{\mathrm{tgt}}$ on h1-walk task. FLAC maintains high performance across a wide range of budgets, indicating robustness. (b) Efficacy of automatic Lagrangian tuning on h1-run (left) and h1-walk (right). Evolution of $\log\alpha$ during training shows a "decrease-then-increase" pattern, indicating that FLAC automatically relaxes constraints for early learning and tightens them later to enforce exploration.
  • Figure 4: Task Domain Visualizations.
  • Figure 5: Sensitivity to NFE. Increasing NFE accelerates early convergence but has little impact on final performance.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Proposition 1: Optimal GSB Solution
  • proof
  • Proposition 2: Convergence of Policy Evaluation
  • Proposition 3: Monotonic Improvement
  • proof