Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAGE: Tool-Augmented LLM Task Solving Strategies in Scalable Multi-Agent Environments

Robert K. Strehlow, Tobias Küster, Oskar F. Kupke, Brandon Llanque Kurps, Fikret Sivrikaya, Sahin Albayrak

TL;DR

The paper addresses the challenge of enabling LLMs to reliably use external tools in dynamic, real-world environments. It introduces SAGE, a tool-augmented conversational interface built on the OPACA multi-agent framework, and details four prompting strategies that vary in architecture and parallelism to generate and execute tool calls in a zero-shot setting. The contributions include a comprehensive design space (design goals, tool-call process, architecture, multi-modality, security), four prompting methods, and an extensive evaluation using a custom benchmark of simulated tool services, highlighting strengths and trade-offs in accuracy and efficiency. The work demonstrates practical implications for scalable, cross-domain LLM-enabled systems, showing how modular tool orchestration can reduce integration overhead while supporting multiple model backends; the open-source implementation further enables adoption and extension in real deployments.

Abstract

Large language models (LLMs) have proven to work well in question-answering scenarios, but real-world applications often require access to tools for live information or actuation. For this, LLMs can be extended with tools, which are often defined in advance, also allowing for some fine-tuning for specific use cases. However, rapidly evolving software landscapes and individual services require the constant development and integration of new tools. Domain- or company-specific tools can greatly elevate the usefulness of an LLM, but such custom tools can be problematic to integrate, or the LLM may fail to reliably understand and use them. For this, we need strategies to define new tools and integrate them into the LLM dynamically, as well as robust and scalable zero-shot prompting methods that can make use of those tools in an efficient manner. In this paper, we present SAGE, a specialized conversational AI interface, based on the OPACA framework for tool discovery and execution. The integration with OPACA makes it easy to add new tools or services for the LLM to use, while SAGE itself presents rich extensibility and modularity. This not only provides the ability to seamlessly switch between different models (e.g. GPT, LLAMA), but also to add and select prompting methods, involving various setups of differently prompted agents for selecting and executing tools and evaluating the results. We implemented a number of task-solving strategies, making use of agentic concepts and prompting methods in various degrees of complexity, and evaluated those against a comprehensive set of benchmark services. The results are promising and highlight the distinct strengths and weaknesses of different task-solving strategies. Both SAGE and the OPACA framework, as well as the different benchmark services and results, are available as Open Source/Open Data on GitHub.

SAGE: Tool-Augmented LLM Task Solving Strategies in Scalable Multi-Agent Environments

TL;DR

The paper addresses the challenge of enabling LLMs to reliably use external tools in dynamic, real-world environments. It introduces SAGE, a tool-augmented conversational interface built on the OPACA multi-agent framework, and details four prompting strategies that vary in architecture and parallelism to generate and execute tool calls in a zero-shot setting. The contributions include a comprehensive design space (design goals, tool-call process, architecture, multi-modality, security), four prompting methods, and an extensive evaluation using a custom benchmark of simulated tool services, highlighting strengths and trade-offs in accuracy and efficiency. The work demonstrates practical implications for scalable, cross-domain LLM-enabled systems, showing how modular tool orchestration can reduce integration overhead while supporting multiple model backends; the open-source implementation further enables adoption and extension in real deployments.

Abstract

Large language models (LLMs) have proven to work well in question-answering scenarios, but real-world applications often require access to tools for live information or actuation. For this, LLMs can be extended with tools, which are often defined in advance, also allowing for some fine-tuning for specific use cases. However, rapidly evolving software landscapes and individual services require the constant development and integration of new tools. Domain- or company-specific tools can greatly elevate the usefulness of an LLM, but such custom tools can be problematic to integrate, or the LLM may fail to reliably understand and use them. For this, we need strategies to define new tools and integrate them into the LLM dynamically, as well as robust and scalable zero-shot prompting methods that can make use of those tools in an efficient manner. In this paper, we present SAGE, a specialized conversational AI interface, based on the OPACA framework for tool discovery and execution. The integration with OPACA makes it easy to add new tools or services for the LLM to use, while SAGE itself presents rich extensibility and modularity. This not only provides the ability to seamlessly switch between different models (e.g. GPT, LLAMA), but also to add and select prompting methods, involving various setups of differently prompted agents for selecting and executing tools and evaluating the results. We implemented a number of task-solving strategies, making use of agentic concepts and prompting methods in various degrees of complexity, and evaluated those against a comprehensive set of benchmark services. The results are promising and highlight the distinct strengths and weaknesses of different task-solving strategies. Both SAGE and the OPACA framework, as well as the different benchmark services and results, are available as Open Source/Open Data on GitHub.
Paper Structure (28 sections, 6 figures, 2 tables)

This paper contains 28 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. The OPACA Multi-Agent Framework
  4. Agent Container Development
  5. Related Work
  6. Concept
  7. Design Goals
  8. Tool Call Generation Process
  9. Architecture
  10. Multi-Modality
  11. Security and Session Handling
  12. Different Methods for LLM Prompting
  13. Simple
  14. Simple-Tools
  15. Tool-Chain
  16. ...and 13 more sections

Figures (6)

  • Figure 1: SAGE System Architecture
  • Figure 2: Screenshot of SAGE, showing 'agents' panel and empty chat window with example prompts.
  • Figure 3: Authentication to Runtime Platform and Agent Container
  • Figure 4: Overview of the Method Architectures implementing the different modules of the Tool Generation Process in \ref{['sec:toolgenprocess']} in their LLM modules. (Red: Planning, Blue: Construction, Green: Evaluation, Yellow: Output Generation.)
  • Figure 5: Comparison of tool usage across different benchmark types.
  • ...and 1 more figures