Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning

TL;DR

The paper tackles the open-ended nature of scientific reasoning by shifting from static tool retrieval to Test-Time Tool Evolution (TTE), where agents synthesize, verify, and refine executable primitives during inference. It introduces a five-module architecture that decomposes problems, retrieves or generates tools, refines atomic primitives, and executes tool-enabled reasoning, with two instantiations: TTE-Zero and TTE-Adapt. The SciEvo benchmark (1,590 tasks, 925 evolved tools) enables rigorous evaluation, where TTE achieves state-of-the-art accuracy and high tool reuse, and demonstrates robust cross-domain adaptation. The work highlights the practicality and challenges of dynamic tool evolution for scalable scientific AI, including latency, safety, and coding-capability considerations, and points toward future directions in governance, robustness, and multi-modal extensions.

Abstract

The central challenge of AI for Science is not reasoning alone, but the ability to create computational methods in an open-ended scientific world. Existing LLM-based agents rely on static, pre-defined tool libraries, a paradigm that fundamentally fails in scientific domains where tools are sparse, heterogeneous, and intrinsically incomplete. In this paper, we propose Test-Time Tool Evolution (TTE), a new paradigm that enables agents to synthesize, verify, and evolve executable tools during inference. By transforming tools from fixed resources into problem-driven artifacts, TTE overcomes the rigidity and long-tail limitations of static tool libraries. To facilitate rigorous evaluation, we introduce SciEvo, a benchmark comprising 1,590 scientific reasoning tasks supported by 925 automatically evolved tools. Extensive experiments show that TTE achieves state-of-the-art performance in both accuracy and tool efficiency, while enabling effective cross-domain adaptation of computational tools. The code and benchmark have been released at https://github.com/lujiaxuan0520/Test-Time-Tool-Evol.

Figures (11)

• Figure 1: Paradigm comparison: Static Tool Paradigm (left) vs Test-Time Tool Evolution (right). Static approaches require pre-collected tool libraries, limiting coverage and domain adaptability. Our test-time evolution starts with an empty library and generates tools on-demand during problem-solving, enabling continuous evolution to new domains and problems.
• Figure 2: The architecture of the Test-Time Tool Evolution (TTE) framework. The system operates through a closed-loop workflow comprising five integrated stages. (1) Structured Task Decomposition: The Problem Analyzer decomposes complex scientific queries into a sequence of executable sub-goals. (2) Dynamic Tool Retrieval: The system queries the Dynamic Tool Registry for existing atomic tools. If retrieval fails, it triggers (3) Generative Tool Synthesis: The Tool Synthesizer creates candidate tools on-the-fly, which undergo strict verification by the Tool Verifier. (4) Atomic Tool Refinement: Validated tools are decoupled into reusable atomic units by the Atomic Decomposer, filtered by the Redundancy Checker, and registered to update the library. (5) Runtime Execution Engine: Once the required tools are successfully retrieved or generated for all the steps, the Tool Executor executes the sequence to synthesize the final answer.
• Figure 3: Tool distribution of the curated SciEvo benchmark. SciEvo covers 25 sub-disciplines across four major scientific fields: Physics (499 tools), Chemistry (192), Mathematics (171), and Materials (63), demonstrating comprehensive coverage of diverse scientific computational needs.
• Figure 4: Accuracy comparison on SciEvo. We compare the "No Tool call" baseline against our TTE-Zero method using direct queries ("Q + Tools") and Sub-goal Decomposition ("S + Tools").
• Figure 5: The prompt used by the Problem Analyzer to decompose user queries into structured execution plans.