Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proximal Policy Optimization in Path Space: A Schrödinger Bridge Perspective

Yuehu Gong, Zeyuan Wang, Yulin Chen, Yanwei Fu

Abstract

On-policy reinforcement learning with generative policies is promising but remains underexplored. A central challenge is that proximal policy optimization (PPO) is traditionally formulated in terms of action-space probability ratios, whereas diffusion- and flow-based policies are more naturally represented as trajectory-level generative processes. In this work, we propose GSB-PPO, a path-space formulation of generative PPO inspired by the Generalized Schrödinger Bridge (GSB). Our framework lifts PPO-style proximal updates from terminal actions to full generation trajectories, yielding a unified view of on-policy optimization for generative policies. Within this framework, we develop two concrete objectives: a clipping-based objective, GSB-PPO-Clip, and a penalty-based objective, GSB-PPO-Penalty. Experimental results show that while both objectives are compatible with on-policy training, the penalty formulation consistently delivers better stability and performance than the clipping counterpart. Overall, our results highlight path-space proximal regularization as an effective principle for training generative policies with PPO.

Proximal Policy Optimization in Path Space: A Schrödinger Bridge Perspective

Abstract

On-policy reinforcement learning with generative policies is promising but remains underexplored. A central challenge is that proximal policy optimization (PPO) is traditionally formulated in terms of action-space probability ratios, whereas diffusion- and flow-based policies are more naturally represented as trajectory-level generative processes. In this work, we propose GSB-PPO, a path-space formulation of generative PPO inspired by the Generalized Schrödinger Bridge (GSB). Our framework lifts PPO-style proximal updates from terminal actions to full generation trajectories, yielding a unified view of on-policy optimization for generative policies. Within this framework, we develop two concrete objectives: a clipping-based objective, GSB-PPO-Clip, and a penalty-based objective, GSB-PPO-Penalty. Experimental results show that while both objectives are compatible with on-policy training, the penalty formulation consistently delivers better stability and performance than the clipping counterpart. Overall, our results highlight path-space proximal regularization as an effective principle for training generative policies with PPO.
Paper Structure (18 sections, 24 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 18 sections, 24 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. On Policy Reinforcement Learning
  4. Generative Policies in Reinforcement Learning
  5. On-Policy Generative Reinforcement Learning
  6. Preliminaries
  7. On Policy Reinforcement Learning
  8. Generalized Schrödinger Bridge
  9. Method
  10. Path Space Formulation of Generative PPO
  11. GSB Inspired PPO with Clipping
  12. GSB Inspired PPO with Penalty
  13. Experiments
  14. Experimental Setup
  15. Main Results on 10 Playground Environments
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Comparison between GSB-PPO and PPO on the 10 playground environments. We report step-return curves over training.
  • Figure 2: Comparison between GSB-PPO and FPO on the 10 playground environments. We report step-return curves over training.
  • Figure 3: KL ablation on CheetahRun.