Training-Free Group Relative Policy Optimization

TL;DR

This paper introduces Training-Free Group Relative Policy Optimization (Training-Free GRPO), a non-parametric approach that improves LLM agent performance in specialized domains without updating model parameters. By collecting and distilling experiential knowledge into a token prior stored in an external library, the method uses multi-epoch, group-based rollouts to curate semantic advantages that guide outputs during inference. It demonstrates strong gains on mathematical reasoning (AIME benchmarks) and web searching (WebWalkerQA) with minimal data and cost, often outperforming fine-tuned 32B models and costly RL baselines. The approach preserves generalization across domains and offers a scalable, cost-effective path for deploying robust LLM agents in real-world settings.

Abstract

Recent advances in Large Language Model (LLM) agents have demonstrated their promising general capabilities. However, their performance in specialized real-world domains often degrades due to challenges in effectively integrating external tools and specific prompting strategies. While methods like agentic reinforcement learning have been proposed to address this, they typically rely on costly parameter updates, for example, through a process that uses Supervised Fine-Tuning (SFT) followed by a Reinforcement Learning (RL) phase with Group Relative Policy Optimization (GRPO) to alter the output distribution. However, we argue that LLMs can achieve a similar effect on the output distribution by learning experiential knowledge as a token prior, which is a far more lightweight approach that not only addresses practical data scarcity but also avoids the common issue of overfitting. To this end, we propose Training-Free Group Relative Policy Optimization (Training-Free GRPO), a cost-effective solution that enhances LLM agent performance without any parameter updates. Our method leverages the group relative semantic advantage instead of numerical ones within each group of rollouts, iteratively distilling high-quality experiential knowledge during multi-epoch learning on a minimal ground-truth data. Such knowledge serves as the learned token prior, which is seamlessly integrated during LLM API calls to guide model behavior. Experiments on mathematical reasoning and web searching tasks demonstrate that Training-Free GRPO, when applied to DeepSeek-V3.1-Terminus, significantly improves out-of-domain performance. With just a few dozen training samples, Training-Free GRPO outperforms fine-tuned small LLMs with marginal training data and cost.

Figures (14)

• Figure 1: Applying Training-Free GRPO on both prompting (without tools) and ReAct react (with tools) achieve improved Mean@32 on AIME benchmarks AIME with DeepSeek-V3.1-Terminus deepseekv3. It consumes significantly fewer training data and lower costs on the 671B LLM than fine-tuning a 32B model retool, serving as a cost-effective alternative to RL methods.
• Figure 2: Comparison of vanilla GRPO and Training-Free GRPO.
• Figure 3: Example of a Training-Free GRPO learning step.
• Figure 4: Statistics at each Training-Free GRPO step with tool use and DeepSeek-V3.1-Terminus.
• Figure 5: Example trajectory without experience in Math Reasoning.