Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges

TL;DR

This work introduces DialogTool, a multi-turn benchmark for stateful tool use in dialogue by modeling the full lifecycle of tool interactions across tool creation, utilization, and role-consistent response. It couples DialogTool with VirtualMobile to simulate API calls within a diverse set of Apps and APIs, enabling end-to-end evaluation of six tasks across three stages on 13 open- and closed-source LLMs. The study reveals that even strong models struggle with long-horizon tool use, particularly in tool creation and execution, and that hierarchical app-then-API selection can improve performance. The findings highlight fundamental challenges in state tracking, argument formatting, and role-consistent response generation, underscoring the need for improved tooling, evaluation protocols, and potentially agent-level workflows to advance practical tool use in multi-turn settings.

Abstract

Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose \texttt{DialogTool}, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) \textit{tool creation}; 2) \textit{tool utilization}: tool awareness, tool selection, tool execution; and 3) \textit{role-consistent response}: response generation and role play. Furthermore, we build \texttt{VirtualMobile} -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs\footnote{We will use tools and APIs alternatively, there are no significant differences between them in this paper.}. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons.

Figures (7)

• Figure 1: A typical example to show the entire life cycle of stateful tool use in multi-turn dialogues. The dialogue agent need to create the tools first or on the fly ①, and then decide whether or not use tools ②, which tool to use ③, execute it with all required arguments fulfilled ④, convert the tool results into responses with different role configs as conversion goes ⑤⑥.
• Figure 2: The pipeline of DialogTool collection by 1) Setting shift: transfer the setting of existing dialogue datasets; 2) Role Play: then rewriting the utterances from system side into role-consistent utterances.
• Figure 3: The three primary errors at tool execution.
• Figure 4: The performance of two prompting strategies.
• Figure 5: This is a sample JSON configuration of Rents App which contains 3 distinct APIs. We also provide name, format and possible values for categorical arguments. In this app, the base required arguments are empty.