Reinforcement Learning for Long-Horizon Interactive LLM Agents

TL;DR

This paper shows that reinforcement learning can effectively train long-horizon interactive LLM agents directly in their target, stateful environments. It introduces LOOP, a memory-efficient PPO variant with a Leave-One-Out advantage that avoids a value network and reuses off-policy rollouts, achieving state-of-the-art results on the AppWorld benchmark with a 32B model. LOOP markedly improves task success versus prior open- and closed-weight baselines and demonstrates robust, diverse behavioral improvements such as API documentation querying, reduced assumption-making, and better error recovery. The results suggest RL on limited data can yield strong generalization in complex, multi-app interactive settings, highlighting LOOP’s practicality for real-world IDA development. A key contribution is the demonstration that token-level PPO with LO0P advantages can sequence thousands of API calls across multiple apps efficiently, paving the way for more capable, memory-conscious LLM-driven agents.

Abstract

Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests. While IDAs powered by instruction-tuned large language models (LLMs) can react to feedback from interface invocations in multi-step exchanges, they have not been trained in their respective digital environments. Prior methods accomplish less than half of tasks in sophisticated benchmarks such as AppWorld. We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments. We formalize this training as a partially observable Markov decision process and derive LOOP, a data- and memory-efficient variant of proximal policy optimization. LOOP uses no value network and maintains exactly one copy of the underlying LLM in memory, making its implementation straightforward and as memory-efficient as fine-tuning a single LLM. A 32-billion-parameter agent trained with LOOP in the AppWorld environment outperforms the much larger OpenAI o1 agent by 9 percentage points (15% relative). To our knowledge, this is the first reported application of RL to IDAs that interact with a stateful, multi-domain, multi-app environment via direct API calls. Our analysis sheds light on the effectiveness of RL in this area, showing that the agent learns to consult the API documentation, avoid unwarranted assumptions, minimize confabulation, and recover from setbacks.

Figures (7)

• Figure 1: An interactive digital agent receives a user request and iteratively interacts with APIs through a Python read-eval-print loop (REPL) to accomplish the task.
• Figure 2: Top: modeling interactive agents as a POMDP. States take the form $[\mathbf{s}_0, \mathbf{c}, x_{1:t}]$, where $\mathbf{s}_0$ is the hidden state (containing the initial state of the coding environment), $\mathbf{c}$ is the initial context for the task (green), and $x_{1:t}$ combines tokens emitted by the LLM (blue) with tokens emitted by the environment (red). Bottom: PPO with a per-token, -turn, or -trajectory importance weights.
• Figure 3: (a) Aggregate changes in agent behavior between the base model and LOOP, averaged over three i.i.d. rollouts per dev (validation) task. (b) Rollouts of the base and LOOP agents on a single dev task. The agent must identify the user's roommates (via phone.search_contacts) and pending requests to them (via venmo.show_sent_payment_requests), and then call venmo.remind_payment_request on each. The base agent assumes the identities of the roommates, leading to task failure. The LOOP agent correctly gathers this information and consistently reads API documentation.
• Figure 4: A visualization of 100 i.i.d. rollouts of an agent on the same task after LOOP training. Each column represents a rollout, containing 12 to 33 color-coded API calls. The majority of rollouts (98 out of 100) successfully complete the task, yet 94 out of 98 exhibit unique sequences of API calls. Even at the high level, behavior does not collapse into a single mode but instead exhibits multiple distinct strategies. The heatmap summarizes the task 2a163ab_1 (\ref{['sec:rollout_2a163ab_1_late']}) where the agent needs to 'like' transactions involving their roommates using the venmo.like_transaction API. Solutions follow four distinct strategies. Mode 1: Directly searching for roommate contacts via phone.search_contacts. Mode 2: Browsing the Venmo social feed first via venmo.show_social_feed to identify relevant transactions. Mode 3: Querying all contact relationships in the phonebook first with phone.contact_relationships. Mode 4: Utilizing venmo.show_social_feed followed by phone.contact_relationships. This diversity is likely key to the success of RL in this domain. It fosters exploration during early training, and prevents severe overfitting late in training.
• Figure 5: Changes in behaviors, aggregated over all 72 tasks used in reinforcement learning training (3 i.i.d. rollouts per task). Showing that the emergent behaviors observed on dev tasks (\ref{['fig:analysis_combined']}) also apply to the train tasks.