Simulating Complex Multi-Turn Tool Calling Interactions in Stateless Execution Environments

TL;DR

This work introduces DiGiT-TC, a data generation framework that creates complex multi-turn tool-calling dialogues in stateless execution environments by flipping the generation order: tool calls are generated first and then the corresponding user requests. It employs plan distillation to prune tool-call sequences, request generation to craft user utterances, and back translation to ensure fidelity and reduce noise, including an explicit mechanism to include implicit tool calls. The authors validate DiGiT-TC on BFCL-v3 and tau^2-bench benchmarks, achieving strong performance relative to smaller, open-model baselines and competitive results with frontier-model trained data. Ablation studies show that both implicit tool calls and back translation contribute significantly to performance, while the method remains privacy-preserving and scalable. Limitations include reliance on non-stateful prompts and potential challenges when fully leveraging stateful environments.

Abstract

Synthetic data has proven itself to be a valuable resource for tuning smaller, cost-effective language models to handle the complexities of multi-turn tool calling conversations. While many frameworks and systems for producing synthetic multi-turn tool calling data have been proposed, prior works have frequently assumed that any tool calling interactions will take place in an execution environment that maintains state. When such an environment is available, this is advantageous as it allows for the validity of an interaction to be determined by whether or not the state of the execution environment matches to some prespecified objective. Unfortunately, this does not hold in many real-world tool use settings, e.g., in enterprise settings where data security is of the utmost importance or in cases where tool specifications are synthesized from multiple sources. In this work, we address this gap by introducing a data generation method, DiGiT-TC, that is designed to produce tool calling conversations that have the characteristics of conversations generated through search in a stateful environment. The key to our technique lies in a novel generation pattern that allows our approach to implicitly represent certain tool calls in the user request. We validate our approach on standard tool calling benchmarks and demonstrate that, even in stateful problem settings, our approach results in strong performance gains.

Figures (14)

• Figure 1: Two examples of interactions, one where both tool calls are explicit and one that requires an implicit tool call, i.e., check_movie_showtimes
• Figure 2: Conversation generation stages in DiGiT-TC
• Figure 3: Construction of a two-turn conversation
• Figure 4: Example of distillation where singleton elements of the top query are filtered out to yield the bottom query
• Figure 5: Example of a clarification error