ING-VP: MLLMs cannot Play Easy Vision-based Games Yet

TL;DR

The paper introduces ING-VP, a first-of-its-kind interactive game-based vision planning benchmark to probe spatial imagination and multi-step reasoning in multimodal LLMs. By evaluating six deterministic games across 300 levels with image-text and text-only inputs, single- and multi-step reasoning, and with/without-history settings, it reveals substantial gaps between state-of-the-art MLLMs and human performance, with average accuracies around 3–4%. The results highlight core bottlenecks in perceiving spatial relationships and maintaining consistent planning, even for top models like Claude-3.5 Sonnet, and show that thinking step-by-step can sometimes hinder rather than help. ING-VP provides a reusable framework and open data/code to guide future advances in visual planning for MLLMs.

Abstract

As multimodal large language models (MLLMs) continue to demonstrate increasingly competitive performance across a broad spectrum of tasks, more intricate and comprehensive benchmarks have been developed to assess these cutting-edge models. These benchmarks introduce new challenges to core capabilities such as perception, reasoning, and planning. However, existing multimodal benchmarks fall short in providing a focused evaluation of multi-step planning based on spatial relationships in images. To bridge this gap, we present ING-VP, the first INteractive Game-based Vision Planning benchmark, specifically designed to evaluate the spatial imagination and multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games, encompassing 300 levels, each with 6 unique configurations. A single model engages in over 60,000 rounds of interaction. The benchmark framework allows for multiple comparison settings, including image-text vs. text-only inputs, single-step vs. multi-step reasoning, and with-history vs. without-history conditions, offering valuable insights into the model's capabilities. We evaluated numerous state-of-the-art MLLMs, with the highest-performing model, Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the anticipated standard. This work aims to provide a specialized evaluation framework to drive advancements in MLLMs' capacity for complex spatial reasoning and planning. The code is publicly available at https://github.com/Thisisus7/ING-VP.git.

Figures (12)

• Figure 1: The overview of ING-VP benchmark.ING-VP comprises 6 distinct games, conducts 3 comparative analyses across 6 experimental settings, and evaluates 5 key capabilities of MLLMs. Additionally, it offers a highly efficient interactive environment for both inference and analysis.
• Figure 2: ING-VP examples sampled from each game. Includes pictures and text representations of Sokoban, Maze, Sudoku, 8-queens, Tower of Hanoi, and 15-puzzle.
• Figure 3: Error distribution over Claude-3.5 Sonnet's 555 errors across different tasks and settings.
• Figure 4: Maze level accuracy of Claude-3.5 Sonnet and GPT-4o across 4 difficulty levels.
• Figure 5: An example showcasing Claude 3.5-Sonnet with a fixed output paradigm.