TUMS: Enhancing Tool-use Abilities of LLMs with Multi-structure Handlers

TL;DR

This paper tackles the limitation of LLMs in tool usage arising from coarse, tool-level parameter generation. It introduces TUMS, a four-module framework (intent recognizer, task decomposer, subtask processor with multi-structure handlers, and executor) that shifts parameter generation from the tool level to the subtasks, enabling tailored parameter construction for tools of varying complexity. Empirical results on the ToolQA benchmark show that TUMS surpasses strong baselines, with significant gains on hard tasks and clear advantages from using multi-structure parameter generation and an intent recognizer. The work demonstrates the feasibility and benefits of parameter-level processing for tool-augmented LLMs and provides ablations that illuminate the contributions of each component, offering a path toward more autonomous, capable tool-enabled agents. Overall, TUMS advances the state of tool-use in LLMs by aligning parameter generation with task and tool difficulty, improving accuracy, efficiency, and reliability in multi-tool scenarios.

Abstract

Recently, large language models(LLMs) have played an increasingly important role in solving a wide range of NLP tasks, leveraging their capabilities of natural language understanding and generating. Integration with external tools further enhances LLMs' effectiveness, providing more precise, timely, and specialized responses. However, LLMs still encounter difficulties with non-executable actions and improper actions, which are primarily attributed to incorrect parameters. The process of generating parameters by LLMs is confined to the tool level, employing the coarse-grained strategy without considering the different difficulties of various tools. To address this issue, we propose TUMS, a novel framework designed to enhance the tool-use capabilities of LLMs by transforming tool-level processing into parameter-level processing. Specifically, our framework consists of four key components: (1) an intent recognizer that identifies the user's intent to help LLMs better understand the task; (2) a task decomposer that breaks down complex tasks into simpler subtasks, each involving a tool call; (3) a subtask processor equipped with multi-structure handlers to generate accurate parameters; and (4) an executor. Our empirical studies have evidenced the effectiveness and efficiency of the TUMS framework with an average of 19.6\% and 50.6\% improvement separately on easy and hard benchmarks of ToolQA, meanwhile, we demonstrated the key contribution of each part with ablation experiments, offering more insights and stimulating future research on Tool-augmented LLMs.

Figures (5)

• Figure 1: The illustration of Tool-level and Parameter-level. The classical tool-use workflow on QA tasks only reaches the depth of the tool level, while our method, TUMS, has a fine-grained treatment on the parameter level.
• Figure 2: The proposed framework: TUMS.
• Figure 3: Comparison of TUMS Variants. TUMS is our proposed method; TUMS-NIR indicates TUMS without intent recognizer; TUMS-OS is the same as TUMS except it has only one-structure handlers.
• Figure 4: The analysis of efficiency. The total cost on the left y-axis(bar chart) and the average cost for one correct answer on the right y-axis(line chart), against different methods on the x-axis.
• Figure 5: The prompts for all modules and two hints examples.