Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs

TL;DR

The paper tackles exploration collapse in RL-enhanced LLM reasoning by shifting regularization from token-level diversity to rollout-set strategy diversity. It introduces Uniqueness-Aware Reinforcement Learning, which uses an LLM judge to cluster rollouts by high-level solution strategies and reweights advantages inversely by cluster size to reward rare yet correct approaches. Empirically, the method improves pass@k and AUC@K across math, physics, and medicine benchmarks, while sustaining exploration and increasing coverage of human-referenced solution strategies. The approach yields stronger performance at larger sampling budgets and provides a practical path toward more diverse and creative reasoning in large language models.

Abstract

Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.

Figures (4)

• Figure 1: Method pipeline for Uniqueness-Aware RL. Given a training problem, we sample multiple rollouts and compute group-normalized GRPO advantages from verifiable rewards. An LLM judge groups rollouts that share the same high-level solution strategy, producing a partition and cluster sizes. We then form uniqueness-weighted advantages, allocating more learning signal to correct but rare strategies and preventing strategy collapse.
• Figure 2: Pass@$k$ accuracy on math, physics, and medicine benchmarks.
• Figure 3: Entropy dynamics under RL. Actor entropy loss over training steps for Qwen2.5, Qwen3, and Olmo3. GRPO exhibits a consistent downward trend, while our uniqueness-aware training maintains a higher and more stable entropy loss.
• Figure 4: Solution Diversity Coverage (cover@32) on AIME. Nodes are distinct human solution ideas. The baseline instruct model (blue dashed) concentrates on standard, low-complexity approaches, while our trained model (red solid) expands the explored region to recover rarer, higher-insight strategies (e.g., Symmedian Similarity; trail/flow viewpoints).