Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SwissNYF: Tool Grounded LLM Agents for Black Box Setting

Somnath Sendhil Kumar, Dhruv Jain, Eshaan Agarwal, Raunak Pandey

TL;DR

This paper addresses the challenge of tool planning under black-box conditions where API outputs, implementations, or secrets are inaccessible. It introduces SwissNYF, a modular framework that generates pseudo-function signatures (CodeSynth), retrieves tools via a corpus, and employs TOPGUN for fast, code-driven planning, supported by a Reflexion-based verifier and a parser to produce executable solution trajectories. Empirical results show TOPGUN outperforms traditional gray-box baselines in win-rate and efficiency, and CodeSynth yields progressively higher-quality function-signatures, demonstrating strong generalization to unseen tools. The work demonstrates a practical, privacy-conscious approach to deploying LLM-enabled agents in complex, irreversible, or latency-sensitive API environments, and provides public code to foster further development.

Abstract

While Large Language Models (LLMs) have demonstrated enhanced capabilities in function-calling, these advancements primarily rely on accessing the functions' responses. This methodology is practical for simpler APIs but faces scalability issues with irreversible APIs that significantly impact the system, such as a database deletion API. Similarly, processes requiring extensive time for each API call and those necessitating forward planning, like automated action pipelines, present complex challenges. Furthermore, scenarios often arise where a generalized approach is needed because algorithms lack direct access to the specific implementations of these functions or secrets to use them. Traditional tool planning methods are inadequate in these cases, compelling the need to operate within black-box environments. Unlike their performance in tool manipulation, LLMs excel in black-box tasks, such as program synthesis. Therefore, we harness the program synthesis capabilities of LLMs to strategize tool usage in black-box settings, ensuring solutions are verified prior to implementation. We introduce TOPGUN, an ingeniously crafted approach leveraging program synthesis for black box tool planning. Accompanied by SwissNYF, a comprehensive suite that integrates black-box algorithms for planning and verification tasks, addressing the aforementioned challenges and enhancing the versatility and effectiveness of LLMs in complex API interactions. The public code for SwissNYF is available at https://github.com/iclr-dummy-user/SwissNYF.

SwissNYF: Tool Grounded LLM Agents for Black Box Setting

TL;DR

This paper addresses the challenge of tool planning under black-box conditions where API outputs, implementations, or secrets are inaccessible. It introduces SwissNYF, a modular framework that generates pseudo-function signatures (CodeSynth), retrieves tools via a corpus, and employs TOPGUN for fast, code-driven planning, supported by a Reflexion-based verifier and a parser to produce executable solution trajectories. Empirical results show TOPGUN outperforms traditional gray-box baselines in win-rate and efficiency, and CodeSynth yields progressively higher-quality function-signatures, demonstrating strong generalization to unseen tools. The work demonstrates a practical, privacy-conscious approach to deploying LLM-enabled agents in complex, irreversible, or latency-sensitive API environments, and provides public code to foster further development.

Abstract

While Large Language Models (LLMs) have demonstrated enhanced capabilities in function-calling, these advancements primarily rely on accessing the functions' responses. This methodology is practical for simpler APIs but faces scalability issues with irreversible APIs that significantly impact the system, such as a database deletion API. Similarly, processes requiring extensive time for each API call and those necessitating forward planning, like automated action pipelines, present complex challenges. Furthermore, scenarios often arise where a generalized approach is needed because algorithms lack direct access to the specific implementations of these functions or secrets to use them. Traditional tool planning methods are inadequate in these cases, compelling the need to operate within black-box environments. Unlike their performance in tool manipulation, LLMs excel in black-box tasks, such as program synthesis. Therefore, we harness the program synthesis capabilities of LLMs to strategize tool usage in black-box settings, ensuring solutions are verified prior to implementation. We introduce TOPGUN, an ingeniously crafted approach leveraging program synthesis for black box tool planning. Accompanied by SwissNYF, a comprehensive suite that integrates black-box algorithms for planning and verification tasks, addressing the aforementioned challenges and enhancing the versatility and effectiveness of LLMs in complex API interactions. The public code for SwissNYF is available at https://github.com/iclr-dummy-user/SwissNYF.
Paper Structure (30 sections, 1 equation, 8 figures, 4 tables, 2 algorithms)

This paper contains 30 sections, 1 equation, 8 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Formulation
  4. Code Generation
  5. SwissNYF
  6. Overview
  7. Function Signature Generation
  8. CodeSynth
  9. Corpus and Retriever
  10. Planner
  11. TOPGUN
  12. Verifier
  13. Parser
  14. Experiments
  15. Gray Box Evaluation
  16. ...and 15 more sections

Figures (8)

  • Figure 1: Illustration of different settings that an LLMs may require to manipulate tools.
  • Figure 2: Illustration of SwissNYF pipeline for tool usage in Black Box setting.
  • Figure 3: Detailed pipeline of our proposed approach with TOPGUN in SwissNYF
  • Figure 4: Illustration of pseudo function and tool planning generated by CodeSynth and TOPGUN, respectively.
  • Figure 5: Illustartion of Self-Reflection Mechanism in TOPGUN
  • ...and 3 more figures