Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent

Bowei Xia, Mengkang Hu, Shijian Wang, Jiarui Jin, Wenxiang Jiao, Yuan Lu, Kexin Li, Ping Luo

TL;DR

This work proposes Tool-Genesis, a diagnostic benchmark designed to quantify agent capabilities across multiple dimensions, including interface compliance, functional correctness, and downstream utility, and finds that even state-of-the-art models struggle to produce precise tool interfaces or executable logic in a one-shot setting.

Abstract

Research on self-evolving language agents has accelerated, drawing increasing attention to their ability to create, adapt, and maintain tools from task requirements. However, existing benchmarks predominantly rely on predefined specifications, which limits scalability and hinders truly autonomous evolution. While recent studies attempt to dynamically generate tools, they primarily emphasize downstream performance, resulting in a "black-box" evaluation that makes it difficult to attribute failures to specific causes. To address this, we propose Tool-Genesis, a diagnostic benchmark designed to quantify agent capabilities across multiple dimensions, including interface compliance, functional correctness, and downstream utility. Tool-Genesis evaluates whether agents can construct task-relevant tools solely from abstract requirements (without preset specifications) and use them to solve realistic problems. Crucially, we find that even state-of-the-art models struggle to produce precise tool interfaces or executable logic in a one-shot setting. These minor initial flaws are amplified through the pipeline, leading to a sharp degradation in downstream metrics. We hope Tool-Genesis will guide future research toward training and steering models to synthesize persistent, general-purpose tools that better address real-world challenges.

Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent

TL;DR

This work proposes Tool-Genesis, a diagnostic benchmark designed to quantify agent capabilities across multiple dimensions, including interface compliance, functional correctness, and downstream utility, and finds that even state-of-the-art models struggle to produce precise tool interfaces or executable logic in a one-shot setting.

Abstract

Research on self-evolving language agents has accelerated, drawing increasing attention to their ability to create, adapt, and maintain tools from task requirements. However, existing benchmarks predominantly rely on predefined specifications, which limits scalability and hinders truly autonomous evolution. While recent studies attempt to dynamically generate tools, they primarily emphasize downstream performance, resulting in a "black-box" evaluation that makes it difficult to attribute failures to specific causes. To address this, we propose Tool-Genesis, a diagnostic benchmark designed to quantify agent capabilities across multiple dimensions, including interface compliance, functional correctness, and downstream utility. Tool-Genesis evaluates whether agents can construct task-relevant tools solely from abstract requirements (without preset specifications) and use them to solve realistic problems. Crucially, we find that even state-of-the-art models struggle to produce precise tool interfaces or executable logic in a one-shot setting. These minor initial flaws are amplified through the pipeline, leading to a sharp degradation in downstream metrics. We hope Tool-Genesis will guide future research toward training and steering models to synthesize persistent, general-purpose tools that better address real-world challenges.
Paper Structure (52 sections, 17 equations, 7 figures, 2 tables)

This paper contains 52 sections, 17 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Formalization
  3. Task Definition
  4. Metrics
  5. Dataset Construction
  6. Compliant MCP-server Data Collection
  7. High-Quality Task & Trajectory Generation
  8. Comprehensive Unit Test Generation
  9. Manual Quality Inspection
  10. Data Analysis
  11. Experiments
  12. Experimental Setup
  13. Experimental Results
  14. Analysis
  15. Signal--validation alignment under closed-loop
  16. ...and 37 more sections

Figures (7)

  • Figure 1: Comparison of tool creation paradigms: (a) Outcome-Driven: Ad-hoc solving with disposable scripts; (b) Code-Centric: Spec-based translation with limited safety; (c) Tool-Genesis(Ours): Inductive design for verified, reusable assets.
  • Figure 2: Dataset construction pipeline of Tool Genesis.
  • Figure 3: Comparison of benchmarks in terms of task reasoning depth and tool compositionality.
  • Figure 4: Overview statistics of Tool Genesis. Left: functional domain coverage of MCP servers across 24 domain classes. Right: dataset scale and task/trajectory structure statistics.
  • Figure 5: Direct prompting vs. code-agent repair on Qwen3 scales: (a) Signal–Validation Alignment (SVA) of Schema-F1 and UT (soft/hard); (b) stage-wise cumulative pass-through across verification stages (Start, L1–L4) for selected scales, with solid/dashed lines denoting the two paradigms.
  • ...and 2 more figures