Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation

Yinuo Liu, Zi Qian, Heng Zhou, Jiahao Zhang, Yajie Zhang, Zhihang Li, Mengyu Zhou, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang

Abstract

Interleaved text-and-image generation represents a significant frontier for Multimodal Large Language Models (MLLMs), offering a more intuitive way to convey complex information. Current paradigms rely on either image generation or retrieval augmentation, yet they typically treat the two as mutually exclusive paths, failing to unify factuality with creativity. We argue that the next milestone in this field is Agentic Tool Planning, where the model serves as a central controller that autonomously determines when, where, and which tools to invoke to produce interleaved responses for visual-critical queries. To systematically evaluate this paradigm, we introduce ATP-Bench, a novel benchmark comprising 7,702 QA pairs (including 1,592 VQA pairs) across eight categories and 25 visual-critical intents, featuring human-verified queries and ground truths. Furthermore, to evaluate agentic planning independent of end-to-end execution and changing tool backends, we propose a Multi-Agent MLLM-as-a-Judge (MAM) system. MAM evaluates tool-call precision, identifies missed opportunities for tool use, and assesses overall response quality without requiring ground-truth references. Our extensive experiments on 10 state-of-the-art MLLMs reveal that models struggle with coherent interleaved planning and exhibit significant variations in tool-use behavior, highlighting substantial room for improvement and providing actionable guidance for advancing interleaved generation. Dataset and code are available at https://github.com/Qwen-Applications/ATP-Bench.

ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation

Abstract

Interleaved text-and-image generation represents a significant frontier for Multimodal Large Language Models (MLLMs), offering a more intuitive way to convey complex information. Current paradigms rely on either image generation or retrieval augmentation, yet they typically treat the two as mutually exclusive paths, failing to unify factuality with creativity. We argue that the next milestone in this field is Agentic Tool Planning, where the model serves as a central controller that autonomously determines when, where, and which tools to invoke to produce interleaved responses for visual-critical queries. To systematically evaluate this paradigm, we introduce ATP-Bench, a novel benchmark comprising 7,702 QA pairs (including 1,592 VQA pairs) across eight categories and 25 visual-critical intents, featuring human-verified queries and ground truths. Furthermore, to evaluate agentic planning independent of end-to-end execution and changing tool backends, we propose a Multi-Agent MLLM-as-a-Judge (MAM) system. MAM evaluates tool-call precision, identifies missed opportunities for tool use, and assesses overall response quality without requiring ground-truth references. Our extensive experiments on 10 state-of-the-art MLLMs reveal that models struggle with coherent interleaved planning and exhibit significant variations in tool-use behavior, highlighting substantial room for improvement and providing actionable guidance for advancing interleaved generation. Dataset and code are available at https://github.com/Qwen-Applications/ATP-Bench.

Paper Structure

This paper contains 20 sections, 1 equation, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Formulation
  4. ATP-Bench
  5. Query Collection
  6. Ground Truth Collection
  7. MAM: Multi-Agent MLLM-as-a-judge
  8. Experiment
  9. Setup
  10. Evaluation Metrics
  11. Results
  12. Ablation Study
  13. Discussion
  14. Conclusion
  15. Full Dataset Statistics
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Examples of eight task categories and corresponding model performance. Since our benchmark targets agentic tool-planning ability, evaluation outputs and ground truths contain tool-planning tags instead of rendered images (See Figure \ref{['fig:overview']}). For better interpretability, we present results obtained after end-to-end tool execution.
  • Figure 2: Overview of ATP-Bench dataset construction and evaluation pipelines.
  • Figure 3: Tool call number distribution for representative models.
  • Figure 4: Tool Adoption Rates per query for representative models. The reference tool is omitted due to its high dominance; all models utilize it in over 90% of queries.
  • Figure 5: Tool Success Rates per tool call for representative models on each tool.
  • ...and 1 more figures