Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missions

TL;DR

The paper tackles the problem that existing benchmarks largely assess LLM-based agents in single-mission settings, failing to reflect real-world dynamic demands. It introduces the Multi-Mission Tool Bench, a framework with four action-types and a mission-switching space, plus a five-role data-generation pipeline to simulate related multi-mission dialogues and evaluate agents with dynamic decision trees. The study provides a comprehensive evaluation across 24 models, revealing significant robustness gaps in multi-mission contexts, especially for complex tool-invocation scenarios and related missions, while also highlighting the benefits of long-term memory in mission relationships. The work offers a practical benchmark and evaluation methodology that informs the development of more robust LLM agents capable of dynamic, multi-mission tool use in real-world settings.

Abstract

Large language models (LLMs) demonstrate strong potential as agents for tool invocation due to their advanced comprehension and planning capabilities. Users increasingly rely on LLM-based agents to solve complex missions through iterative interactions. However, existing benchmarks predominantly access agents in single-mission scenarios, failing to capture real-world complexity. To bridge this gap, we propose the Multi-Mission Tool Bench. In the benchmark, each test case comprises multiple interrelated missions. This design requires agents to dynamically adapt to evolving demands. Moreover, the proposed benchmark explores all possible mission-switching patterns within a fixed mission number. Specifically, we propose a multi-agent data generation framework to construct the benchmark. We also propose a novel method to evaluate the accuracy and efficiency of agent decisions with dynamic decision trees. Experiments on diverse open-source and closed-source LLMs reveal critical factors influencing agent robustness and provide actionable insights to the tool invocation society.

Figures (23)

• Figure 2: Visualization of mission switching space. a) Four distinct colors represents four different action-types. The green dot indicates the agent sequentially selects four type of actions to execute four missions. b) The distribution of the proposed benchmark within the mission switching space. Each row corresponds to a different number of missions. Each dot indicates a specific combination of the current and preceding action-types. Colored dots indicate combinations included in the benchmark, while gray dots indicate their absence. c) Distribution of four other agent benchmarks in the space.
• Figure 3: The multi-agent framework.
• Figure 4: The dependencies among tools.
• Figure 5: Visualization of the dynamic decision tree during evaluation.
• Figure 6: Overall accuracy of agents on the whole benchmark.