Can GRPO Help LLMs Transcend Their Pretraining Origin?

TL;DR

This work analyzes Reinforcement Learning with Verifiable Rewards (RLVR) via GRPO to determine when it helps LLM reasoning and generalization. It provides a theoretical characterization showing GRPO performs a conservative reweighting of the base model, and conducts controlled, synthetic-data experiments across reasoning depth, input length, token representation, and compositionality to probe OOD generalization. The findings reveal that GRPO improves OOD performance only when the target task aligns with pretrained biases, and gains on in-distribution tasks diminish as accuracy saturates; token- and depth-based generalization require prior exposure to analogous patterns. The study reframes GRPO as a bias-sharpening tool rather than a universal solver and motivates algorithms that expand the model’s solution space beyond its pretraining origin, with implications for how to select pretraining data and augmentation strategies in practice.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), primarily driven by the Group Relative Policy Optimization (GRPO) algorithm, is a leading approach for enhancing the reasoning abilities of Large Language Models (LLMs). Despite its wide adoption, GRPO's gains are often inconsistent; for instance, a model may show significant improvement in one reasoning domain, like mathematics, yet remain stagnant in another, such as medicine. This inconsistency raises a critical question: under what conditions does GRPO improve reasoning and generalize out-of-distribution (OOD)? We investigate this from a data distribution perspective. We first prove theoretically that GRPO is a conservative reweighting scheme, bounded by the base model's distribution and thus unable to discover completely novel solutions. We further validate this in carefully designed controlled studies by training transformers from scratch, evaluating generalization across reasoning depth, input length, token representation, and compositionality. Our results provide a principled explanation for GRPO's boundaries: OOD improvement emerges only when the target task aligns with the model's pretrained biases, while gains on in-distribution (ID) tasks diminish as performance saturates. This reframes GRPO not as a universal reasoning enhancer but as a tool that sharpens pretraining biases. Our findings motivate future development of algorithms that can expand a model's capabilities beyond its pretraining origin.

Figures (9)

• Figure 1: Accuracy on various datasets after applying GRPO on LLMs. We benchmark GRPO on four LLMs (Qwen2.5-3B, Qwen2.5-7B qwen2025qwen25technicalreport, Llama3.2-3B-Instruct, Llama-3.1-8B-Instruct grattafiori2024llama3herdmodels) across four datasets (Math500 hendrycks2021math, MedMCQA pmlr-v174-pal22a, PharmDB abdullahi2025kpaths, CommonsenseQA talmor2019commonsenseqa). The results show that Qwen models achieve larger gains on math and general-domain datasets, whereas Llama models achieves larger gains on general and clinical-domain datasets. Notably, Qwen2.5-7B improves by $31.2\%$ on Math500 but only $5.2\%$ on PharmDB; Llama-3.1-8B-Instruct gains $16.7\%$ on PharmDB but merely $1.4\%$ on Math500. These findings highlight that GRPO’s effectiveness is highly dependent on the specific model–dataset pairing.
• Figure 2: Our overall framework to probe the generalization limits of GRPO. We train transformers from scratch with synthetic data and evaluate generalization in four settings: (1) reasoning depth, (2) input length, (3) token representation, and (4) compositional reasoning.
• Figure 3: Reasoning depth generalization. Top row (a–c): models pretrained on shallow reasoning (1–2 steps, ID) mixed with deeper reasoning (3 steps, OOD). Bottom row (d–f): models pretrained on deep reasoning (2–3 steps, ID) mixed with shallower reasoning (1 step, OOD).
• Figure 4: Input length generalization. Top row (a–c): models pretrained on shorter sequences (5–6 chars, ID) mixed with longer OOD sequences (7 chars, OOD). Bottom row (d–f): models pretrained on longer sequences (6–7 chars, ID) mixed with shorter OOD sequences (5 chars, OOD).
• Figure 5: Token representation generalization.