RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models

TL;DR

RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries, transforms reward acquisition into a dynamic tool synthesis and invocation task, demonstrating superior generalization through dynamic reward orchestration.

Abstract

Large language model alignment via reinforcement learning depends critically on reward function quality. However, static, domain-specific reward models are often costly to train and exhibit poor generalization in out-of-distribution scenarios encountered during RL iterations. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize programmatic verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. Experimental results demonstrate that RLAR yields consistent performance gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks. On RewardBench-V2, RLAR significantly outperforms static baselines and approaches the performance upper bound, demonstrating superior generalization through dynamic reward orchestration. The data and code are available on this link: https://github.com/ZhuoerFeng/RLAR.

Figures (7)

• Figure 1: A conceptual comparison between traditional reward modeling and our proposed RLAR framework.
• Figure 2: The Reinforce Learning with Agentic Reward (RLAR) Framework.
• Figure 3: Ablation studies on Llama-3.1 and Qwen3.
• Figure 4: Matching tools with source training dataset distribution.
• Figure 5: Maximum Advantage Estimations