MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models

TL;DR

MultiVerse introduces a large-scale, multi-turn vision-language benchmark to evaluate VLMs across a broad spectrum of tasks and image domains. It combines an instance-specific checklist evaluation using GPT-4o with a rich dataset (647 dialogues, 484 tasks, 8 interaction goals, 25 image domains) to provide robust assessment beyond prior datasets. The experiments show that even strong models struggle with sustained multi-turn interactions, and in-context dialogue history significantly improves performance for smaller models, illustrating the value of context and scaling. The work also demonstrates a practical evaluation protocol with attention to per-turn reasoning across 37 aspects and minimal verbosity bias. Overall, MultiVerse offers a comprehensive platform for diagnosing multi-turn interaction capabilities in VLMs and guiding future model improvements.

Abstract

Vision-and-Language Models (VLMs) have shown impressive capabilities on single-turn benchmarks, yet real-world applications often demand more intricate multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only partially capture the breadth and depth of conversational scenarios encountered by users. In this work, we introduce MultiVerse, a novel multi-turn conversation benchmark featuring 647 dialogues - each averaging four turns - derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484 tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from factual knowledge and perception to advanced reasoning tasks such as mathematics and coding. To facilitate robust assessment, we propose a checklist-based evaluation method that leverages GPT-4o as the automated evaluator, measuring performance across 37 key aspects, including perceptual accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve only a 50% success rate in complex multi-turn conversations, highlighting the dataset's challenging nature. Notably, we find that providing full dialogue context significantly enhances performance for smaller or weaker models, emphasizing the importance of in-context learning. We believe MultiVerse is a landscape of evaluating multi-turn interaction abilities for VLMs.

Figures (12)

• Figure 1: Examples of multi-turn conversations from MMDU liu2024mmdu and ConvBench liu2024convbench, along with their limitations. The graph below compares MMDU, ConvBench, and MultiVerse in terms of query lexical diversity, the number of code/math tasks in responses, and the total number of tasks.
• Figure 2: An overview of MultiVerse construction pipeline, consisting of five steps: (1) Source Image Collection, gathering images from various benchmarks; (2) Personal Background Generation, creating user personas with specific goals; (3) Multi-Turn Conversation Generation, where an AI assistant analyzes model performance based on evaluation tables; (4) Manual Reviewing, assessing response correctness and relevance; and (5) Checklist Generation, ensuring quality and coherence in AI responses.
• Figure 3: Detailed distribution of (a) Task, (b) Interaction Goal, and (c) Image Domain in MultiVerse.
• Figure 4: Correlation between the checklist completion ratio and quality assessment. The red line represents the best-fit linear regression, indicating a strong positive correlation $(R^2=0.44)$ between the two metrics.
• Figure 5: Average performance of 18 VLMs under two different evaluation settings: Oracle and Self-Prediction.