Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs

TL;DR

Graph2Eval addresses the challenge of evaluating autonomous multimodal agents beyond static datasets by grounding task generation in a knowledge-graph framework. It defines a five-stage pipeline (data parsing, knowledge-graph construction, subgraph sampling, task generation, and coverage optimization) and uses task templates and meta-paths to instantiate document comprehension and web interaction tasks. The authors implement Graph2Eval-Bench, comprising 1,319 tasks, and demonstrate efficient generation alongside clear differentiation of agent/model performance across scales and settings. The work provides a scalable, reproducible approach for measuring reasoning, collaboration, and interactive capabilities, with future directions including safety task generation and detailed error attribution.

Abstract

As multimodal LLM-driven agents continue to advance in autonomy and generalization, evaluation based on static datasets can no longer adequately assess their true capabilities in dynamic environments and diverse tasks. Existing LLM-based synthetic data methods are largely designed for LLM training and evaluation, and thus cannot be directly applied to agent tasks that require tool use and interactive capabilities. While recent studies have explored automatic agent task generation with LLMs, most efforts remain limited to text or image analysis, without systematically modeling multi-step interactions in web environments. To address these challenges, we propose Graph2Eval, a knowledge graph-based framework that automatically generates both multimodal document comprehension tasks and web interaction tasks, enabling comprehensive evaluation of agents' reasoning, collaboration, and interactive capabilities. In our approach, knowledge graphs constructed from multi-source external data serve as the task space, where we translate semantic relations into structured multimodal tasks using subgraph sampling, task templates, and meta-paths. A multi-stage filtering pipeline based on node reachability, LLM scoring, and similarity analysis is applied to guarantee the quality and executability of the generated tasks. Furthermore, Graph2Eval supports end-to-end evaluation of multiple agent types (Single-Agent, Multi-Agent, Web Agent) and measures reasoning, collaboration, and interaction capabilities. We instantiate the framework with Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning document comprehension and web interaction scenarios. Experiments show that Graph2Eval efficiently generates tasks that differentiate agent and model performance, revealing gaps in reasoning, collaboration, and web interaction across different settings and offering a new perspective for agent evaluation.

Figures (9)

• Figure 2: Overview of the dataset generated by Graph2Eval.
• Figure 3: Workflow for dataset generation in Graph2Eval: (1) Data Ingestion (Top Left / Right): parsing documents and crawling web pages to extract structured content. (2) Knowledge Graph Construction (Middle Left): building the graph by identifying nodes and edges that encode semantic, structural, and interactive relations. (3) Subgraph Sampling (Middle Right): applying scenario-specific sampling strategies for document and web tasks based on execution modes. (4) Task Generation (Bottom Right): instantiating and composing tasks from sampled subgraphs, producing diverse, executable task units. (5) Coverage Optimization (Bottom Left): evaluating and selecting generated tasks to ensure quality, diversity, and representativeness.
• Figure 4: Coverage proportions of Web and Doc task dimensions used in the task optimization.
• Figure 5: Comparison of Processing Times Across Documents and Websites.
• Figure 6: The heatmap shows the LLM evaluation quality scores of model responses across different task types (X-axis) and models (Y-axis).