Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

Minghan Chen, Guikun Chen, Wenguan Wang, Yi Yang

TL;DR

SEED-GRPO addresses prompt-level uncertainty in LLM-based policy optimization by introducing semantic entropy to quantify meaning-level disagreement among multiple model outputs. This uncertainty signal modulates the per-prompt advantages in a PPO-style GRPO framework, downweighting updates for high-uncertainty prompts to stabilize learning and enable curriculum-like progression. Empirical results across five mathematical reasoning benchmarks show state-of-the-art average accuracy with a 7B model and demonstrate robust gains through ablations on weighting, clustering, rollout count, and base models. The work highlights uncertainty-aware policy optimization as a practical mechanism to improve reasoning tasks while maintaining computational efficiency.

Abstract

Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variation reflects LLM's uncertainty about the input prompt, a signal of how confidently the model understands a given problem. However, vanilla Group Relative Policy Optimization (GRPO) treats all prompts equally during policy updates, ignoring this important information about the model's knowledge boundaries. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs' uncertainty of the input prompts semantic entropy. Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates. This uncertainty-aware training mechanism enables dynamic adjustment of policy update magnitudes based on question uncertainty. It allows more conservative updates on high-uncertainty questions while maintaining the original learning signal on confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization.

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

TL;DR

SEED-GRPO addresses prompt-level uncertainty in LLM-based policy optimization by introducing semantic entropy to quantify meaning-level disagreement among multiple model outputs. This uncertainty signal modulates the per-prompt advantages in a PPO-style GRPO framework, downweighting updates for high-uncertainty prompts to stabilize learning and enable curriculum-like progression. Empirical results across five mathematical reasoning benchmarks show state-of-the-art average accuracy with a 7B model and demonstrate robust gains through ablations on weighting, clustering, rollout count, and base models. The work highlights uncertainty-aware policy optimization as a practical mechanism to improve reasoning tasks while maintaining computational efficiency.

Abstract

Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variation reflects LLM's uncertainty about the input prompt, a signal of how confidently the model understands a given problem. However, vanilla Group Relative Policy Optimization (GRPO) treats all prompts equally during policy updates, ignoring this important information about the model's knowledge boundaries. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs' uncertainty of the input prompts semantic entropy. Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates. This uncertainty-aware training mechanism enables dynamic adjustment of policy update magnitudes based on question uncertainty. It allows more conservative updates on high-uncertainty questions while maintaining the original learning signal on confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization.
Paper Structure (14 sections, 9 equations, 2 figures, 4 tables)

This paper contains 14 sections, 9 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SEED-GRPO: Uncertainty-Aware Policy Optimization
  4. Motivation: Uncertainty-Aware Learning
  5. SEED-GRPO Illustration via Math Reasoning Example
  6. SEED-GRPO Illustration via Math Reasoning Example
  7. Discussion and Analysis
  8. Experiments
  9. Experiments
  10. Experimental Setup
  11. Quantitative Comparison Results
  12. Ablation Study
  13. Limitation and Future Work
  14. Conclusion

Figures (2)

  • Figure 1: Intuitive explanation of semantic entropy. For Question 1, although the 6 responses have slight syntactic variations, 5 of them convey the same meaning, indicating low semantic entropy and high model certainty. For Question 2, the 6 responses can be clustered into 6 distinct meaning classes, resulting in high semantic entropy and indicating significant model uncertainty.
  • Figure 2: The SEED-GRPO framework incorporating semantic entropy for uncertainty-aware reinforcement learning. The framework samples multiple responses from a pre-trained LLM, computes semantic entropy to measure model uncertainty, and modulates the advantage function accordingly to enable more conservative updates for high-uncertainty questions.