GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

TL;DR

GameVerse is presented, a comprehensive video game benchmark that enables a reflective visual interaction loop and uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies.

Abstract

Human gameplay is a visually grounded interaction loop in which players act, reflect on failures, and watch tutorials to refine strategies. Can Vision-Language Models (VLMs) also learn from video-based reflection? We present GameVerse, a comprehensive video game benchmark that enables a reflective visual interaction loop. Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies. To facilitate systematic and scalable evaluation, we also introduce a cognitive hierarchical taxonomy spanning 15 globally popular games, dual action space for both semantic and GUI control, and milestone evaluation using advanced VLMs to quantify progress. Our experiments show that VLMs benefit from video-based reflection in varied settings, and perform best by combining failure trajectories and expert tutorials-a training-free analogue to reinforcement learning (RL) plus supervised fine-tuning (SFT).Our project page is available at https://gameverse-bench.github.io/ . Our code is available at https://github.com/THUSI-Lab/GameVerse .

Figures (71)

• Figure 1: Humans improve gameplay by reflecting on failures and consulting expert tutorials. Our benchmark mimics this process, enabling agents to learn from video-based reflection. Humans show the largest gains, while all models benefit to varying degrees.
• Figure 2: Overview of GameVerse, which effectively probes the capability boundaries of VLMs in diverse video game worlds. GameVerse supports dual action space, enables human-like reflection by integrating failure and tutorial videos, and delivers process score.
• Figure 3: Top: Average performance across 5 cognitive and 3 difficulty categories. Bottom: Average improvement across models.
• Figure 4: Left: Average improvement across 5 cognitive category tasks. Right: Overall improvement statistics of 120 trials.
• Figure 5: Performance of 4 VLMs on 2 games of GameVerse in semantic action mode with three latency settings. Reactive model is average performance of GPT-4o and Qwen3-VL-8B. Reasoning model is average performance of Seed-1.8 and Gemini-2.5-Pro.