Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering

TL;DR

This work addresses the inefficiency and sparsity of external verifiers in reinforcement learning from human feedback (RLHF) for large language models by introducing Latent-GRPO. It leverages intrinsic rewards derived from the geometry of latent space, specifically the clustering of terminal token representations around a truth centroid, to guide policy optimization via GRPO without external supervision. The central method, Iterative Robust Centroid Estimation (IRCE), projects terminal states onto a unit sphere, iteratively computes a robust centroid with soft Gaussian weights, and yields dense, normalized rewards based on distances to the centroid. Empirical results across GSM8K, MATH, Open-Platypus and model scales demonstrate about a $2\times$ training speedup, strong generalization to unseen tasks, and robustness, supporting the potential of latent-geometry-based self-verification for scalable RLHF.

Abstract

Group Relative Policy Optimization (GRPO) significantly enhances the reasoning performance of Large Language Models (LLMs). However, this success heavily relies on expensive external verifiers or human rules. Such dependency not only leads to significant computational costs and training latency, but also yields sparse rewards that hinder optimization efficiency. To address these challenges, we propose Latent-GRPO, a framework that derives intrinsic rewards directly from latent space geometry. Crucially, our empirical analysis reveals a compelling geometric property: terminal token representations of correct reasoning trajectories form dense clusters with high intra-class similarity, whereas incorrect trajectories remain scattered as outliers. In light of this discovery, we introduce the Iterative Robust Centroid Estimation (IRCE) algorithm, which generates dense, continuous rewards by mitigating magnitude fluctuations via spherical projection and estimating a robust ``truth centroid'' through iterative aggregation. Experimental results on multiple datasets show that our method maintains model performance while achieving a training speedup of over 2x compared to baselines. Furthermore, extensive results demonstrate strong generalization ability and robustness. The code will be released soon.

Figures (6)

• Figure 1: Comparison between conventional GRPO and Latent-GRPO. Conventional GRPO relies on expensive external verifiers to compute rewards, whereas Latent-GRPO autonomously extracts reward signals from the geometric structure of the latent space, eliminating external dependencies.
• Figure 2: 2D PCA projection of 1,000 rollouts. Correct trajectories (green) form a dense consensus core around the truth centroid (gold star), while incorrect ones (red) scatter as outliers.
• Figure 3: Overview of the Latent-GRPO framework. The policy model generates a group of responses for each prompt. Instead of relying on external verifiers, we extract the hidden states of the last token from each trajectory and apply the Iterative Robust Centroid Estimation (IRCE) algorithm to compute intrinsic rewards based on geometric clustering in the latent space. These rewards are then used to compute group-relative advantages for policy optimization. The entire process operates within the latent space, achieving zero additional inference overhead while providing dense reward signals.
• Figure 4: Validation of geometric-based scoring. (a) Distribution separability across quality levels (correct, partial, incorrect). (b-d) Group-level ranking consistency in representative 8-trajectory groups.
• Figure 5: Visualization of latent manifold consensus across various model scales and benchmarks. Each panel shows the 2D PCA projection of 1,000 terminal hidden states. (a-d) consistently demonstrate that correct trajectories (green) cluster into a dense consensus core, while incorrect ones (red) are scattered.