DeepAgent: A General Reasoning Agent with Scalable Toolsets

TL;DR

DeepAgent tackles the gap between large reasoning models and real-world task autonomy by enabling autonomous thinking, tool discovery, and action within a single reasoning process. It introduces memory folding to compress long interaction histories into structured episodic, working, and tool memories, reducing token growth and error accumulation. ToolPO provides end-to-end RL training with an LLM-simulated API and fine-grained credit assignment to stabilize learning and improve tool use. Across eight benchmarks spanning general tool-use and downstream applications, DeepAgent consistently outperforms baselines, especially in open-set retrieval, illustrating scalable, real-world applicability.

Abstract

Large reasoning models have demonstrated strong problem-solving abilities, yet real-world tasks often require external tools and long-horizon interactions. Existing agent frameworks typically follow predefined workflows, which limit autonomous and global task completion. In this paper, we introduce DeepAgent, an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution within a single, coherent reasoning process. To address the challenges of long-horizon interactions, particularly the context length explosion from multiple tool calls and the accumulation of interaction history, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories, reducing error accumulation while preserving critical information. To teach general-purpose tool use efficiently and stably, we develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens. Extensive experiments on eight benchmarks, including general tool-use tasks (ToolBench, API-Bank, TMDB, Spotify, ToolHop) and downstream applications (ALFWorld, WebShop, GAIA, HLE), demonstrate that DeepAgent consistently outperforms baselines across both labeled-tool and open-set tool retrieval scenarios. This work takes a step toward more general and capable agents for real-world applications. The code and demo are available at https://github.com/RUC-NLPIR/DeepAgent.

Figures (5)

• Figure 1: Overall performance on (a) general tool usage tasks and (b) downstream applications (best score as 100%).
• Figure 2: Comparison of agent paradigms: (a) Traditional agents with predefined workflows, (b) Deep Research agents that can autonomously call limited tools, and (c) Our DeepAgent, a fully autonomous reasoning agent that dynamically discovers and invokes helpful tools, all within a continuous agentic reasoning process.
• Figure 3: Overview of the DeepAgent framework. The main reasoning model autonomously discovers tools, executes actions, and folds previous memory to restart with structured memories, all within a unified thinking process. The DeepAgent is trained end-to-end with ToolPO, an RL method that uses a tool simulator to simulate large-scale real-world tool APIs, and rewards both final task success and correct intermediate tool calls through fine-grained advantage attribution.
• Figure 4: Visualization of training dynamics, including (a) reward scores and (b) validation scores across training steps.
• Figure 5: Scaling analysis of performance with respect to maximum action limits on WebShop and GAIA datasets.