From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIs

TL;DR

This work proposes Sum2Act, a novel framework that augments large language models with open-world API access by incorporating a router and a state manager. At each step, the LLM proposes actions and then summarizes outcomes to update a compact task state, enabling robust handling of real-world, dynamic tool responses. Empirical results on ToolBench show Sum2Act outperforms ReAct and DFSDT in both pass and win metrics, and the approach extends to multimodal tasks by integrating vision APIs for conditional image generation and VQA-driven editing. The method offers a practical pathway toward more capable open-world reasoning in LLMs, combining the strengths of chain-of-thought-like prompting with structured stateful control and error reflection.

Abstract

The distinction between humans and animals lies in the unique ability of humans to use and create tools. Tools empower humans to overcome physiological limitations, fostering the creation of magnificent civilizations. Similarly, enabling foundational models like Large Language Models (LLMs) with the capacity to learn external tool usage may serve as a pivotal step toward realizing artificial general intelligence. Previous studies in this field have predominantly pursued two distinct approaches to augment the tool invocation capabilities of LLMs. The first approach emphasizes the construction of relevant datasets for model fine-tuning. The second approach, in contrast, aims to fully exploit the inherent reasoning abilities of LLMs through in-context learning strategies. In this work, we introduce a novel tool invocation pipeline designed to control massive real-world APIs. This pipeline mirrors the human task-solving process, addressing complicated real-life user queries. At each step, we guide LLMs to summarize the achieved results and determine the next course of action. We term this pipeline `from Summary to action', Sum2Act for short. Empirical evaluations of our Sum2Act pipeline on the ToolBench benchmark show significant performance improvements, outperforming established methods like ReAct and DFSDT. This highlights Sum2Act's effectiveness in enhancing LLMs for complex real-world tasks.

Figures (7)

• Figure 1: Overview of the general tool learning framework, which consists of three components: LLM, tool set, and history. The user provides an instruction to the LLM, which iteratively selects tools from the tool set, and records results in the history until it deems the instruction complete. The LLM then responds to the user.
• Figure 2: The Framework of Sum2Act: Components and Workflow. The router decides the next action based on user instruction and the current state, then finds and runs the right API. The State Manager summarizes the observation from APIs and updates the state with information from the API. The process ends when the router decides the goal is reached and provides the answer to the user.
• Figure 3: Case Studies: Utilizing Open-World APIs for Diverse Queries. The left scenario demonstrates querying for C compiler versions and video information, while the right scenario showcases the use of specific APIs to gather Florida's weather forecast and flight data.
• Figure 4: Case Studies: Demonstrating Sum2Act's versatility in vision tasks, showcasing its proficiency in conditional image generation and its ability to integrate multiple vision APIs for targeted image editing.
• Figure 5: Case Studies: Illustrating Sum2Act's adeptness in handling intricate tasks through a combination of open-world and vision APIs. On the left, a webcam image undergoes style transfer, while on the right, an image is generated that corresponds to a searched vegan recipe. This highlights Sum2Act's proficiency in seamlessly integrating both open-world and varied vision APIs.