Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extraction

Zhancun Mu

TL;DR

DeFlow addresses the expressivity-optimization dilemma in offline RL by decoupling behavior modeling from value-based policy refinement. It uses multi-step flow matching to faithfully capture the behavior manifold and a lightweight, instance-aware refinement module to maximize $Q(s,a)$ without backpropagating through ODE solvers, guided by a dynamic constraint on action deviation $\delta$ via a learnable Lagrangian multiplier. The base flow $\mu_\psi$ is trained pure with Flow Matching, while $f_\phi$ optimizes locally within the manifold, enabled by a stop-gradient connection that avoids BPTT through the solver. Empirically, DeFlow achieves state-of-the-art or competitive results on the OGBench and D4RL benchmarks and demonstrates seamless offline-to-online adaptation, with automatic constraint tuning reducing hyperparameter engineering. This decoupled approach preserves manifold geometry, improves optimization stability, and enables robust online fine-tuning, suggesting a practical path for scalable policy extraction from expressive generative policies.

Abstract

We present DeFlow, a decoupled offline RL framework that leverages flow matching to faithfully capture complex behavior manifolds. Optimizing generative policies is computationally prohibitive, typically necessitating backpropagation through ODE solvers. We address this by learning a lightweight refinement module within an explicit, data-derived trust region of the flow manifold, rather than sacrificing the iterative generation capability via single-step distillation. This way, we bypass solver differentiation and eliminate the need for balancing loss terms, ensuring stable improvement while fully preserving the flow's iterative expressivity. Empirically, DeFlow achieves superior performance on the challenging OGBench benchmark and demonstrates efficient offline-to-online adaptation.

DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extraction

TL;DR

DeFlow addresses the expressivity-optimization dilemma in offline RL by decoupling behavior modeling from value-based policy refinement. It uses multi-step flow matching to faithfully capture the behavior manifold and a lightweight, instance-aware refinement module to maximize without backpropagating through ODE solvers, guided by a dynamic constraint on action deviation via a learnable Lagrangian multiplier. The base flow is trained pure with Flow Matching, while optimizes locally within the manifold, enabled by a stop-gradient connection that avoids BPTT through the solver. Empirically, DeFlow achieves state-of-the-art or competitive results on the OGBench and D4RL benchmarks and demonstrates seamless offline-to-online adaptation, with automatic constraint tuning reducing hyperparameter engineering. This decoupled approach preserves manifold geometry, improves optimization stability, and enables robust online fine-tuning, suggesting a practical path for scalable policy extraction from expressive generative policies.

Abstract

We present DeFlow, a decoupled offline RL framework that leverages flow matching to faithfully capture complex behavior manifolds. Optimizing generative policies is computationally prohibitive, typically necessitating backpropagation through ODE solvers. We address this by learning a lightweight refinement module within an explicit, data-derived trust region of the flow manifold, rather than sacrificing the iterative generation capability via single-step distillation. This way, we bypass solver differentiation and eliminate the need for balancing loss terms, ensuring stable improvement while fully preserving the flow's iterative expressivity. Empirically, DeFlow achieves superior performance on the challenging OGBench benchmark and demonstrates efficient offline-to-online adaptation.
Paper Structure (29 sections, 12 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 29 sections, 12 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Offline RL as Constrained Optimization
  4. Critic Learning.
  5. The Actor's Dilemma.
  6. Conditional Flow Matching
  7. Flow as a Policy.
  8. DeFlow: Disentangling Fidelity and Optimality
  9. Revisiting Existing Coupling Paradigms
  10. Paradigm I: End-to-End Policy Steering.
  11. Paradigm II: Single-step Shortcut Distillation.
  12. The DeFlow Approach
  13. Seamless Offline-to-Online (O2O) Adaptation.
  14. Related Works
  15. Policy Optimization with Diffusion and Flow Models
  16. ...and 14 more sections

Figures (3)

  • Figure 1: DeFlow Framework.(a)Iterative Flow Prior: A multi-step flow policy faithfully captures the complex, multimodal structure of the dataset. (b)The Compression Bottleneck (e.g., FQL): Existing methods often distill the flow into a one-step policy to enable joint optimization. This compression leads to expressivity collapse, failing to cover the manifold. (c)DeFlow (Ours): We retain the full expressive power of the iterative flow (Blue) to define the support, and decouple the optimization into a lightweight refinement module (Orange). This enables precise value maximization without destroying the manifold geometry.
  • Figure 2: Visualization of Action Landscapes. Overlay of Q-value contours and action samples. Gray: Base Flow samples (data proxy); Blue: FQL (one-step); Red: DeFlow (ours). Arrows indicate the refinement trajectory. By anchoring optimization to the Base Flow manifold, DeFlow avoids (a) mode collapse and (b) OOD drift, effectively refining actions within the valid support.
  • Figure 3: Offline-to-online RL results ($\mathbf{8}$ seeds).