ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models

TL;DR

ToolDial tackles the gap in realistic tool-using dialogue benchmarks by introducing 11,111 multi-turn dialogues based on RapidAPI APIs, featuring API chaining via an API graph and a 16-action taxonomy. The authors present a four-step data-generation pipeline (graph construction, action sequences, scenario instruction, utterance generation) and evaluate multiple models on dialogue state tracking, action prediction, and faithfulness, revealing substantial room for improvement, especially in long-horizon DST and parameter extraction. Fine-tuning on ToolDial significantly benefits smaller models and improves several hard actions, underscoring the dataset’s value for training TALMs. Overall, ToolDial provides both a rich benchmark and a scalable methodology to study and advance tool-augmented language models in realistic, multi-turn settings, with code and data openly released.

Abstract

Tool-Augmented Language Models (TALMs) leverage external APIs to answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information is missing. To address these limitations, we construct and release ToolDial, a dataset comprising 11,111 multi-turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorporate 16 user and system actions (e.g., "Request", "Clarify", "Fail inform") to capture the rich dynamics of real-world interactions. Second, we simulate dialogues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the required information. To facilitate this process, we introduce a method for generating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to predict correct actions and extract input parameter values for API calls from the dialogue history. Modern language models achieve accuracy scores below 70%, indicating substantial room for improvement. We release our dataset and code at https://github.com/holi-lab/ToolDial.

Figures (5)

• Figure 1: Overall structure of ToolDial. This represents the whole pipeline of our method.
• Figure 2: An example dialogue from ToolDial. This illustrates the user and TALM actions for each turn, along with corresponding utterances. It also shows the reasoning steps TALM undergoes, including API calls and retriever calls, before asking or responding to the user.
• Figure 3: Action graph based on predefined user and system actions. This represents the whole multi turn interaction between user and TALM in our dataset.
• Figure 4: Possible cases of two action sequences according to perform types "Fail inform".
• Figure 5: DST Accuracy for each model as the number of dialogue turns increases.