Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GIPO: Gaussian Importance Sampling Policy Optimization

Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Baigui Sun, Yang Liu

TL;DR

Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency.

Abstract

Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency.

GIPO: Gaussian Importance Sampling Policy Optimization

TL;DR

Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency.

Abstract

Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation. However, RL remains limited by poor data efficiency, particularly in settings where interaction data are scarce and quickly become outdated. To address this challenge, GIPO (Gaussian Importance sampling Policy Optimization) is proposed as a policy optimization objective based on truncated importance sampling, replacing hard clipping with a log-ratio-based Gaussian trust weight to softly damp extreme importance ratios while maintaining non-zero gradients. Theoretical analysis shows that GIPO introduces an implicit, tunable constraint on the update magnitude, while concentration bounds guarantee robustness and stability under finite-sample estimation. Experimental results show that GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes, from near on-policy to highly stale data, while exhibiting superior bias--variance trade-off, high training stability and improved sample efficiency.
Paper Structure (54 sections, 4 theorems, 51 equations, 9 figures, 1 table, 1 algorithm)

This paper contains 54 sections, 4 theorems, 51 equations, 9 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Truncated Off-policy Correction under Policy Lag
  4. Beyond Hard Clipping: Smooth Surrogates and Off-policy PPO
  5. Preliminaries
  6. MDP and Policy Optimization
  7. Replay-Heavy Training and Policy Lag
  8. Standard PPO and Utilization Collapse
  9. Surrogate-side vs. Target-side Correction
  10. Gaussian Importance Sampling Policy Optimization
  11. Log-Space Gaussian Weighting
  12. The GIPO Surrogate Objective
  13. Desirable Properties
  14. Symmetric Trust and Boundedness.
  15. Smoothness.
  16. ...and 39 more sections

Key Result

Lemma 5.1

For any $\tau>0$,

Figures (9)

  • Figure 1: Comparison of gradient weights. Top: Weights vs. importance ratio $\rho$. Bottom: Weights vs. $\log(\rho)$. The log-scale plots highlight GIPO's unique symmetry ($\omega(\rho) = \omega(1/\rho)$) compared to PPO and SAPO, ensuring balanced updates for equivalent deviations.
  • Figure 2: Learning curves on the Meta-World benchmark under Stale and Fresh regimes. Left: Stale; Right: Fresh. Curves show episodic average return over environment steps from a single run per configuration.
  • Figure 3: Learning curves on LIBERO suites under Stale and Fresh regimes. Rows from top to bottom: LIBERO-Object, LIBERO-Spatial, LIBERO-Goal, and LIBERO-10. Left: Stale; Right: Fresh. Curves show episodic average return over environment steps from a single run per configuration.
  • Figure 4: Bias--variance trade-off in 2×2 GridWorld. Case A represents high policy lag, Case B represents moderate policy lag, and Case C represents low policy lag. The dashed line indicates the Pareto frontier derived by GIPO.
  • Figure 5: Training-time evolution of staleness indicators on a representative task. We plot mean version gap, $\mathrm{OldFrac}$, $\mathrm{OldGapP95}$, and mean data age for both regimes. The Stale regime transitions into larger version gaps and higher old-sample fractions, while the Fresh regime remains near the low-lag range throughout training.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Lemma 5.1
  • Theorem 5.2: Lower bound for the expected performance
  • Lemma 5.3: Global bound on $\omega(\bar{\rho}';\sigma)\rho'$
  • Theorem 5.4