ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

TL;DR

ArenaRL reframes open-ended agent training from pointwise scalar rewards to intra-group relative ranking, addressing discriminative collapse by using process-aware pairwise evaluations within a seeded tournament framework. The method combines five tournament topologies, identifying seeded single-elimination as the most efficient and accurate for open-ended tasks, and introduces an adversarial arena with a bidirectional judge to stabilize feedback signals. To support robust evaluation, the authors build Open-Travel and Open-DeepResearch benchmarks that cover SFT, RL, and multi-dimensional assessment, and demonstrate ArenaRL's superior performance across travel planning, deep research, and open-ended writing, including real-world business scenarios. The work yields practical gains in planning robustness, reasoning depth, and task reliability, offering a scalable path for deploying open-ended LLM agents in complex real-world settings.

Abstract

Reinforcement learning has substantially improved the performance of LLM agents on tasks with verifiable outcomes, but it still struggles on open-ended agent tasks with vast solution spaces (e.g., complex travel planning). Due to the absence of objective ground-truth for these tasks, current RL algorithms largely rely on reward models that assign scalar scores to individual responses. We contend that such pointwise scoring suffers from an inherent discrimination collapse: the reward model struggles to distinguish subtle advantages among different trajectories, resulting in scores within a group being compressed into a narrow range. Consequently, the effective reward signal becomes dominated by noise from the reward model, leading to optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation mechanism, employing multi-level rubrics to assign fine-grained relative scores to trajectories. Additionally, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. Empirical results confirm that the built seeded single-elimination scheme achieves nearly equivalent advantage estimation accuracy to full pairwise comparisons with O(N^2) complexity, while operating with only O(N) complexity, striking an optimal balance between efficiency and precision. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we build Open-Travel and Open-DeepResearch, two high-quality benchmarks featuring a comprehensive pipeline covering SFT, RL training, and multi-dimensional evaluation. Extensive experiments show that ArenaRL substantially outperforms standard RL baselines, enabling LLM agents to generate more robust solutions for complex real-world tasks.

Figures (10)

• Figure 1: (a) Illustration of discriminative collapse in pointwise evaluation: We analyse the rewards of trajectory groups generated for a query example during RL training through two evaluation settings. First, the intra-group reward signal represents the pointwise rewards assigned to each trajectory within the group during a single-round evaluation. And the intra-group variance, denoted as $\sigma_{\text{group}}$, quantifies the degree of variation among different trajectories. Secondly, the N-round noise statistic present the average reward and corresponding scoring noise band for each trajectory across N independent evaluation repetitions, from which the noise variance $\sigma_{\text{noise}}$ is estimated. Observations reveal that the evaluation noise variance $\sigma_{\text{noise}}$ is substantial, comparable to the intra-group variance $\sigma_{\text{group}}$. This results in an extremely low signal-to-noise ratio (SNR), causing genuine advantages to be obscured by noise and hindering effective reinforcement learning optimization. (b) ArenaRL Performance: By shifting from pointwise scalar scoring to tournament-based relative ranking, ArenaRL significantly outperforms baselines (SFT, GRPO, GSPO) across diverse open-ended benchmarks.
• Figure 2: The overall of the proposed ArenaRL algorithm. ArenaRL replaces conventional pointwise scalar reward paradigm with intra-group relative ranking and designs five distinct tournament topologies to optimally balance training efficiency against the accuracy of advantage estimation.
• Figure 3: The construction process of Open-Travel and Open-DeepResearch benchmarks.
• Figure 4: (a) The impact of Group Size $N$ on performance of Open-Travel benchmark. (b) The consistency between LLM and human evaluations. (c) The performance trend of ArenaRL in training Qwen3-8b via direct RL without cold start.
• Figure 5: Prompt of Open-Travel task.