Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are Large Vision Language Models Good Game Players?

Xinyu Wang, Bohan Zhuang, Qi Wu

TL;DR

The paper introduces LVLM-Playground, a game-based framework to evaluate Large Vision Language Models on perception, reasoning, decision-making, and adversarial skills using six board games across four tasks. It addresses limitations of traditional LVLM benchmarks by integrating visual inputs with structured game states and automated metrics, and by incorporating a search-based opponent to simulate adversarial play. Empirical results reveal that commercial LVLMs generally excel in perception and rule-following for simple tasks but struggle with dense visual perception, complex rule comprehension, and sustained end-to-end gameplay, often exhibiting stochastic or misaligned behaviors. The framework provides a comprehensive, multimodal method for comparing LVLMs and highlights directions for improving visual-language alignment and long-horizon strategic reasoning in future work.

Abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable abilities in understanding and reasoning about both visual and textual information. However, existing evaluation methods for LVLMs, primarily based on benchmarks like Visual Question Answering and image captioning, often fail to capture the full scope of LVLMs' capabilities. These benchmarks are limited by issues such as inadequate assessment of detailed visual perception, data contamination, and a lack of focus on multi-turn reasoning. To address these challenges, we propose \method{}, a game-based evaluation framework designed to provide a comprehensive assessment of LVLMs' cognitive and reasoning skills in structured environments. \method{} uses a set of games to evaluate LVLMs on four core tasks: Perceiving, Question Answering, Rule Following, and End-to-End Playing, with each target task designed to assess specific abilities, including visual perception, reasoning, decision-making, etc. Based on this framework, we conduct extensive experiments that explore the limitations of current LVLMs, such as handling long structured outputs and perceiving detailed and dense elements. Code and data are publicly available at https://github.com/xinke-wang/LVLM-Playground.

Are Large Vision Language Models Good Game Players?

TL;DR

The paper introduces LVLM-Playground, a game-based framework to evaluate Large Vision Language Models on perception, reasoning, decision-making, and adversarial skills using six board games across four tasks. It addresses limitations of traditional LVLM benchmarks by integrating visual inputs with structured game states and automated metrics, and by incorporating a search-based opponent to simulate adversarial play. Empirical results reveal that commercial LVLMs generally excel in perception and rule-following for simple tasks but struggle with dense visual perception, complex rule comprehension, and sustained end-to-end gameplay, often exhibiting stochastic or misaligned behaviors. The framework provides a comprehensive, multimodal method for comparing LVLMs and highlights directions for improving visual-language alignment and long-horizon strategic reasoning in future work.

Abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable abilities in understanding and reasoning about both visual and textual information. However, existing evaluation methods for LVLMs, primarily based on benchmarks like Visual Question Answering and image captioning, often fail to capture the full scope of LVLMs' capabilities. These benchmarks are limited by issues such as inadequate assessment of detailed visual perception, data contamination, and a lack of focus on multi-turn reasoning. To address these challenges, we propose \method{}, a game-based evaluation framework designed to provide a comprehensive assessment of LVLMs' cognitive and reasoning skills in structured environments. \method{} uses a set of games to evaluate LVLMs on four core tasks: Perceiving, Question Answering, Rule Following, and End-to-End Playing, with each target task designed to assess specific abilities, including visual perception, reasoning, decision-making, etc. Based on this framework, we conduct extensive experiments that explore the limitations of current LVLMs, such as handling long structured outputs and perceiving detailed and dense elements. Code and data are publicly available at https://github.com/xinke-wang/LVLM-Playground.

Paper Structure

This paper contains 23 sections, 26 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation for LVLMs
  4. LVLM for Games
  5. LVLM-Playground
  6. Overview
  7. Game Selection
  8. Abilities
  9. Tasks
  10. Evaluations
  11. Experiments
  12. Results and Findings
  13. Conclusion
  14. Calculation of Ability Ratings
  15. Weighting Coefficients
  16. ...and 8 more sections

Figures (7)

  • Figure 1: Overview of LVLM-Playground. LVLMs receive visual and textual inputs to perform tasks such as Perceiving, Q&A, Rule Following, and End-to-End Playing in game environments. Dashed lines indicate interactions with a search-based opponent using algorithms like Minimax or Alpha-Beta pruning in competitive games.
  • Figure 2: The LVLM-Playground comprises six different games, including Tic Tac Toe, Reversi, Sudoku, Minesweeper, Gomoku, and Chess.
  • Figure 3: Models are evaluated on various games within the LVLM-Playground framework, with each game consisting of four tasks: Perceiving, Q&A, Rule Following, and E2E Playing. Each task targets one or more abilities, including Perception, Reasoning, Decision, and Adversary.
  • Figure 4: Game-weighted performance of LVLMs on four abilities in the LVLM-Playground.
  • Figure 5: Average Human Ranking of Game Difficulty
  • ...and 2 more figures