Evolving from Tool User to Creator via Training-Free Experience Reuse in Multimodal Reasoning

TL;DR

The paper tackles the limitations of fixed, manually crafted tools in tool-integrated reasoning for LLMs and proposes a training-free approach to enable self-evolving agents. It introduces UCT, a three-module architecture comprising Online Task Loop for problem solving with on-demand tool creation, Online Build Loop for iterative tool synthesis and testing, and Offline Memory Consolidation for memory management and tool library upkeep. The authors also release TRBench, a 959-sample multimodal benchmark designed to evaluate tool-use reasoning across math, science, and VQA. Empirical results show substantial cross-domain gains and state-of-the-art performance on TRBench, validating the feasibility and value of training-free experience reuse for continuous tool evolution in open-world tasks.

Abstract

Existing Tool-Integrated Reasoning (TIR) models have effectively extended the question-answering capabilities of LLMs by incorporating external tools. However, real-world scenarios present numerous open-ended problems where fixed tools often fail to meet task requirements. Furthermore, the lack of self-optimization mechanisms means that erroneous tool outputs can mislead the LLM's responses. Additionally, the construction of existing tools entails significant manual effort, which consequently constrains their applicability. Recognizing that the reasoning traces of LLMs encapsulate implicit problem-solving capabilities, we propose UCT, a novel training-free framework that transforms agents from tool users to tool creators. This approach harvests reasoning experiences and distills them into reusable assets. This method transforms the agent from a mere tool user into a tool creator, enabling adaptive tool creation and self-updating during the inference process. We also introduce a memory consolidation mechanism to maintain the tool library, ensuring high reusability of retained experiential memory for subsequent reasoning tasks. This novel automated tool construction paradigm continuously improves tool quality during reasoning, allowing the overall agent system to progress without additional training. Extensive experiments demonstrate that our method serves as a novel paradigm for enhancing the capabilities of TIR models. In particular, the significant performance gains achieved +20.86%$\uparrow$ and +23.04%$\uparrow$ on benchmarks across multi-domain mathematical and scientific reasoning tasks validate the self-evolving capability of the agent.

Figures (6)

• Figure 1: Comparison of tool-creating agents. (a) For this specific math problem, the standard Chain-of-Thought (CoT) cotComT method fails and makes errors even during simple calculations. (b) Previous tool creation methods typically solve problems by generating code specific to the current instance. However, these tools are tailored solely to the immediate problem, making them non-reusable for other tasks and still prone to errors. (c) Ours. We propose a framework capable of reusing tool creation experience. During the inference process for task solving, UCT can utilize, create, and self-evolve existing tools. Furthermore, we design an offline memory consolidation module to generalize tool memory and transform it into reusable tool experience assets.
• Figure 2: The overall architecture of the proposed self-evolving agent framework. The system operates through three coupled phases: (1) The Online Task Loop governs the problem-solving process using the ReAct paradigm. At step $t$, the policy model $\pi_{\theta}$ predicts the optimal action $a_{t+1} = \operatorname*{arg\,max} P_{\theta}(a \mid h_t, o_t, \mathcal{T})$ based on the interaction history $h_t$ and current observation $o_t$. The action space $\mathcal{A}$ dynamically integrates reasoning thoughts, tool execution ($\mathcal{T}_{\text{core}} \cup \mathcal{T}_{\text{cre}}$), and tool creation requests. (2) The Online Build Loop is triggered by a creation ticket $\mathbf{c}_{\text{ticket}}$ to iteratively synthesize new tool code. This isolated refinement process is formalized as $C^{(k)} = \Psi_{\text{build}}(C^{(k-1)}, \mathcal{R}_{\text{critic}}, \mathcal{R}_{\text{sandbox}})$, where the generator optimizes the code $C^{(k)}$ by fusing feedback from the critic model ($\mathcal{R}_{\text{critic}}$) and the sandbox execution environment ($\mathcal{R}_{\text{sandbox}}$). (3) The Offline Memory Consolidation module asynchronously evolves the tool library by merging, classifying, and pruning tool assets to ensure long-term scalability and retrieval efficiency.
• Figure 3: Data distribution of TRBenchmark. TRBench is a multimodal tool-use reasoning benchmark spanning Mathematics, Science, and General Question Answering. It comprises 959 challenging tool reasoning problems organized into 11 sub-categories across 3 major domains.
• Figure 4: The tool library generated by UCT. The library comprises 7 major categories, 64 sub-categories, and 207 specific computational tools. The pie chart illustrates the distribution of these specific tools relative to the total collection, highlighting the richness and hierarchical organization of our generated toolset.
• Figure 5: Tool call rounds and accuracy of UCT.