Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Smooth Gate Functions for Soft Advantage Policy Optimization

Egor Denisov, Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko

TL;DR

An analysis of the findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks provides practical guidance for designing smoother and more robust policy optimization objectives for large language model training.

Abstract

Group Relative Policy Optimization (GRPO) has significantly advanced the training of large language models and enhanced their reasoning capabilities, while it remains susceptible to instability due to the use of hard clipping. Soft Adaptive Policy Optimization (SAPO) addresses this limitation by replacing clipping with a smooth sigmoid-based gate function, which leads to more stable updates. We have decided to push this theory further and investigate the impact of different gate functions on both training stability and final model performance. We formalize the key properties that admissible gates should satisfy and identify several families of such functions for empirical evaluation. This paper presents an analysis of our findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks. These results provide practical guidance for designing smoother and more robust policy optimization objectives for large language model training.

Smooth Gate Functions for Soft Advantage Policy Optimization

TL;DR

An analysis of the findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks provides practical guidance for designing smoother and more robust policy optimization objectives for large language model training.

Abstract

Group Relative Policy Optimization (GRPO) has significantly advanced the training of large language models and enhanced their reasoning capabilities, while it remains susceptible to instability due to the use of hard clipping. Soft Adaptive Policy Optimization (SAPO) addresses this limitation by replacing clipping with a smooth sigmoid-based gate function, which leads to more stable updates. We have decided to push this theory further and investigate the impact of different gate functions on both training stability and final model performance. We formalize the key properties that admissible gates should satisfy and identify several families of such functions for empirical evaluation. This paper presents an analysis of our findings based on experiments conducted with the Qwen2.5-7B-Instruct model on mathematical reasoning tasks. These results provide practical guidance for designing smoother and more robust policy optimization objectives for large language model training.
Paper Structure (13 sections, 9 equations, 1 figure)

This paper contains 13 sections, 9 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Group Relative Policy Optimization (GRPO).
  5. Soft Adaptive Policy Optimization (SAPO).
  6. Methodology
  7. Experiments
  8. Experimental setup
  9. Training.
  10. Reward.
  11. Evaluation.
  12. Results
  13. Conclusion

Figures (1)

  • Figure 1: Temperature-dependent behavior of the considered gate functions (top row) and their gradients (bottom row) for $\tau \in \{1,5,10\}$. All functions are normalized to be positive on $[0; +\infty)$ and to pass through the point $(1,1)$. Increasing the temperature sharpens the transition around $u = 1$, leading to more localized and higher gradient peaks for smooth gates, while the clipped variant exhibits piecewise-linear behavior with constant gradients in its active region.