RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System

TL;DR

RLAnything introduces a dynamic, closed-loop reinforcement learning framework that jointly forges the environment, policy, and reward model to amplify learning signals in long-horizon, agentic scenarios. It integrates policy learning with step-wise reward signals, optimizes the reward model via consistency feedback, and automatically adapts environment tasks using critic signals, backed by theoretical guarantees on reward precision. Empirical results across GUI agents, text-based LLMs, and coding tasks demonstrate improved policy convergence, stronger reward models, and scalable environment expansion, achieving notable gains on OSWorld, AlfWorld, and LiveBench. The framework fosters active learning from experience and suggests a path toward scalable, self-evolving agents in complex real-world settings.

Abstract

We propose RLAnything, a reinforcement learning framework that dynamically forges environment, policy, and reward models through closed-loop optimization, amplifying learning signals and strengthening the overall RL system for any LLM or agentic scenarios. Specifically, the policy is trained with integrated feedback from step-wise and outcome signals, while the reward model is jointly optimized via consistency feedback, which in turn further improves policy training. Moreover, our theory-motivated automatic environment adaptation improves training for both the reward and policy models by leveraging critic feedback from each, enabling learning from experience. Empirically, each added component consistently improves the overall system, and RLAnything yields substantial gains across various representative LLM and agentic tasks, boosting Qwen3-VL-8B-Thinking by 9.1% on OSWorld and Qwen2.5-7B-Instruct by 18.7% and 11.9% on AlfWorld and LiveBench, respectively. We also that optimized reward-model signals outperform outcomes that rely on human labels. Code: https://github.com/Gen-Verse/Open-AgentRL

Figures (12)

• Figure 2: Motivation and takeaways of our RLAnything framework. First, in complex real-world applications, reinforcement learning benefits from integrating step-wise rewards with outcome rewards. Second, the reward model can be jointly optimized with the policy via outcome supervision and self-consistency signals. Third, we show that adapting environment task difficulty to the policy’s capability not only facilitates policy learning but also improves reward model training within our framework. Environment tasks leverage critic feedback from both the policy and the reward model to drive automatic, targeted adaptation, further enabling active learning from experience.
• Figure 3: Examples of environment task adaptation based on critic feedback across computer use agent, text-game agent, and coding LLM in our experiments. The critic feedback is summarized from the reward model’s evaluations and is used to automatically adapt tasks.
• Figure 4: Each dynamic component consistently improves policy's training curve across LLM agent and GUI agent settings.
• Figure 5: Results on OSWorld tasks for different models, including UI-TARS1.5-7B qin2025ui, OpenCUA-7B wang2025opencua, Qwen3-VL-8B-Thinking Qwen3-VL, and our optimized model. Results are averaged over three independent runs, with the maximum number of interaction steps set to 50.
• Figure 6: (a) shows the need for an integrated reward and that our optimized reward model alone provides a stronger learning signal than outcome supervision (the standard GRPO deepseekmath setting). (b) shows scaling with number of interaction steps.

Theorems & Definitions (4)

• Theorem 1
• Theorem 2
• proof : Proof
• proof : Proof