Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRETEL: A Goal-driven Retrieval and Execution-based Trial Framework for LLM Tool Selection Enhancing

Zongze Wu, Yani Guo, Churong Liang, Runnan Li

TL;DR

The paper addresses the semantic-functional gap in tool retrieval for LLM-driven agents and introduces GRETEL, an execution-driven framework that validates candidate tools via a Plan-Execute-Evaluate loop and accumulates execution evidence to produce a functionally grounded re-ranking. By integrating sandboxed trials and an evidence-based holistic re-ranker, GRETEL dramatically improves functional correctness and rank quality on ToolBench compared with semantic baselines and SOTA re-rankers. Core contributions include formalizing the gap, detailing an agentic workflow with trial-based validation, and providing comprehensive empirical validation plus ablation and error analyses. The work has practical significance for building robust, autonomous AI agents capable of reliable tool use in real-world settings.

Abstract

Despite remarkable advances in Large Language Model capabilities, tool retrieval for agent-based systems remains fundamentally limited by reliance on semantic similarity, which fails to capture functional viability. Current methods often retrieve textually relevant but functionally inoperative tools due to parameter mismatches, authentication failures, and execution constraints--a phenomenon we term the semantic-functional gap. We introduce GRETEL, to address this gap through systematic empirical validation. GRETEL implements an agentic workflow that processes semantically retrieved candidates through sandboxed plan-execute-evaluate cycles, generating execution-grounded evidence to distinguish truly functional tools from merely descriptive matches. Our comprehensive evaluation on the ToolBench benchmark demonstrates substantial improvements across all metrics: Pass Rate (at 10) increases from 0.690 to 0.826, Recall (at 10) improves from 0.841 to 0.867, and NDCG (at 10) rises from 0.807 to 0.857.. These results establish that execution-based validation provides a more reliable foundation for tool selection than semantic similarity alone, enabling more robust agent performance in real-world applications.

GRETEL: A Goal-driven Retrieval and Execution-based Trial Framework for LLM Tool Selection Enhancing

TL;DR

The paper addresses the semantic-functional gap in tool retrieval for LLM-driven agents and introduces GRETEL, an execution-driven framework that validates candidate tools via a Plan-Execute-Evaluate loop and accumulates execution evidence to produce a functionally grounded re-ranking. By integrating sandboxed trials and an evidence-based holistic re-ranker, GRETEL dramatically improves functional correctness and rank quality on ToolBench compared with semantic baselines and SOTA re-rankers. Core contributions include formalizing the gap, detailing an agentic workflow with trial-based validation, and providing comprehensive empirical validation plus ablation and error analyses. The work has practical significance for building robust, autonomous AI agents capable of reliable tool use in real-world settings.

Abstract

Despite remarkable advances in Large Language Model capabilities, tool retrieval for agent-based systems remains fundamentally limited by reliance on semantic similarity, which fails to capture functional viability. Current methods often retrieve textually relevant but functionally inoperative tools due to parameter mismatches, authentication failures, and execution constraints--a phenomenon we term the semantic-functional gap. We introduce GRETEL, to address this gap through systematic empirical validation. GRETEL implements an agentic workflow that processes semantically retrieved candidates through sandboxed plan-execute-evaluate cycles, generating execution-grounded evidence to distinguish truly functional tools from merely descriptive matches. Our comprehensive evaluation on the ToolBench benchmark demonstrates substantial improvements across all metrics: Pass Rate (at 10) increases from 0.690 to 0.826, Recall (at 10) improves from 0.841 to 0.867, and NDCG (at 10) rises from 0.807 to 0.857.. These results establish that execution-based validation provides a more reliable foundation for tool selection than semantic similarity alone, enabling more robust agent performance in real-world applications.
Paper Structure (17 sections, 1 equation, 1 figure, 5 tables, 1 algorithm)

This paper contains 17 sections, 1 equation, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tool-Augmented Large Language Models
  4. Semantic-based Tool Retrieval
  5. Execution-based Tool Validation and Planning
  6. Methodology
  7. Problem Formulation
  8. The GRETEL Agentic Workflow
  9. Trial-based Evidence Generation
  10. Holistic Re-ranking with Accumulated Evidence
  11. Experiments
  12. Experimental Setup
  13. Main Quantitative Results
  14. Ablation and SOTA Analysis
  15. Error Analysis: The Semantic-Functional Gap
  16. ...and 2 more sections

Figures (1)

  • Figure 1: GRETEL ingests a semantically retrieved tool list and employs a Plan-Execute-Evaluate loop, yielding a final re-ranking grounded in functional validation.