Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment

TL;DR

This work introduces Interleaved Scene Graph (ISG), a comprehensive evaluation framework for responses that interleave text and images. It formalizes four granularity levels—holistic, structural, block, and image—via a scene-graph representation that links text and image blocks with relations. Complementing ISG is ISG-Bench, a 1,150-sample benchmark spanning 8 categories and 21 subtasks to stress vision-language dependencies and golden-answer evaluation. The authors also propose ISG-Agent, a Plan-Execute-Refine baseline that leverages advanced tools to generate interleaved content, achieving strong performance, particularly on vision-driven tasks. Across experiments, unified models underperform compared with compositional approaches, underscoring the need for targeted interleaved-data and more reliable, interpretable evaluation for multimodal generation.

Abstract

Many real-world user queries (e.g. "How do to make egg fried rice?") could benefit from systems capable of generating responses with both textual steps with accompanying images, similar to a cookbook. Models designed to generate interleaved text and images face challenges in ensuring consistency within and across these modalities. To address these challenges, we present ISG, a comprehensive evaluation framework for interleaved text-and-image generation. ISG leverages a scene graph structure to capture relationships between text and image blocks, evaluating responses on four levels of granularity: holistic, structural, block-level, and image-specific. This multi-tiered evaluation allows for a nuanced assessment of consistency, coherence, and accuracy, and provides interpretable question-answer feedback. In conjunction with ISG, we introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8 categories and 21 subcategories. This benchmark dataset includes complex language-vision dependencies and golden answers to evaluate models effectively on vision-centric tasks such as style transfer, a challenging area for current models. Using ISG-Bench, we demonstrate that recent unified vision-language models perform poorly on generating interleaved content. While compositional approaches that combine separate language and image models show a 111% improvement over unified models at the holistic level, their performance remains suboptimal at both block and image levels. To facilitate future work, we develop ISG-Agent, a baseline agent employing a "plan-execute-refine" pipeline to invoke tools, achieving a 122% performance improvement.

Figures (59)

• Figure 1: An illustration of differences of each generative model performance on (vision-language dominate) tasks, with merely text and image output cannot address the user's problem. See Section \ref{['benchmark']} for how we define (vision dominate) and (language-dominate). Left: Text Generation; Middle: Image Generation; Right: Interleaved Text-and-Image Generation.
• Figure 2: ISG first interprets the user's query into a scene-graph-like structure to enable fine-grained assessment at three levels: 1) At the structural level, ISG predicts the query's interleaved structure; 2) At the block level, nodes represent text-image blocks connected by requirement edges; 3) At the image level, the graph consists of entities, their attributes, and their relationships. Finally, ISG converts each element within the graph structure into questions, evaluates the model's interleaved output using a QA module, and subsequently summarizes these results into a comprehensive assessment.
• Figure 3: Left: An overview of ISG-Bench. Right: Distribution analysis of textual content length and image count for queries and golden answers.
• Figure 4: Distributions of VQA instances in Block-level (Upper) and Image-level (Lower).
• Figure 6: Case study evaluation performed by ISG-Bench, with each generation resulting to a four-level scoring sheet. Mini-GPT5 and Seed-14B fail to generate interleaved content, while Anole generates low-quality images.