VideoGameBench: Can Vision-Language Models complete popular video games?

TL;DR

VideoGameBench introduces a rigorous benchmark for vision-language models to play complete 1990s video games in real time using raw visual input. It emphasizes generalization by including secret games and a Lite variant that paused gameplay to separate reasoning from latency, revealing substantial gaps in current frontier models' planning, memory, and visual understanding. Automated progress tracking via perceptual hashing of walkthrough checkpoints enables granular scoring across many titles. Across GPT-4o, Claude, Gemini, and open-source baselines, the best model achieves under 1% completion, underscoring the challenge of bridging human-like inductive biases with unassisted multimodal agents. The work motivates focused advances in perception, memory, and planning for robust, generalizable video-game-playing AI.

Abstract

Vision-language models (VLMs) have achieved strong results on coding and math benchmarks that are challenging for humans, yet their ability to perform tasks that come naturally to humans--such as perception, spatial navigation, and memory management--remains understudied. Real video games are crafted to be intuitive for humans to learn and master by leveraging innate inductive biases, making them an ideal testbed for evaluating such capabilities in VLMs. To this end, we introduce VideoGameBench, a benchmark consisting of 10 popular video games from the 1990s that VLMs directly interact with in real-time. VideoGameBench challenges models to complete entire games with access to only raw visual inputs and a high-level description of objectives and controls, a significant departure from existing setups that rely on game-specific scaffolding and auxiliary information. We keep three of the games secret to encourage solutions that generalize to unseen environments. Our experiments show that frontier vision-language models struggle to progress beyond the beginning of each game. We find inference latency to be a major limitation of frontier models in the real-time setting; therefore, we introduce VideoGameBench Lite, a setting where the game pauses while waiting for the LM's next action. The best performing model, Gemini 2.5 Pro, completes only 0.48% of VideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalization of the human skills mentioned above into this benchmark motivates progress in these research directions.

Figures (8)

• Figure 1: VideoGameBench provides an environment for vision-language models (VLMs) to interact with video game emulators -- for example, the emulator above is playing The Legend of Zelda: Link's Awakening. Given information about the game controls and emulator and access to the game's raw frames, models provide actions in natural language.
• Figure 2: To track progress on VideoGameBench, we scrape deterministic checkpoints from online walkthroughs of video games and compute image hashes using buchner2013imagehash. These hashes are compared to the current game screen to determine if an agent has reached a checkpoint, and the score is determined based on the relative timestamp of the checkpoint with respect to the walkthrough.
• Figure 3: To determine when a run ends in VideoGameBench, we provide a bound of $20 \times$ the length of the scraped walkthroughs in Table \ref{['tab:game_walkthrough_links']}. For VideoGameBench Lite, we use a similar metric, but determine time by translating one step to one second. Time is in the format (hours:minutes:seconds).
• Figure 4: VideoGameBench features a set of 20 video games from the MS-DOS and Game Boy platforms that VLMs are tasked with playing to completion.
• Figure 5: VideoGameBench checkpoint lengths. We show the length of each game walkthrough and the position of each checkpoint as a black divider. Checkpoints in VideoGameBench are mapped to the timestamp it was scraped from in an online walkthrough video to determine the percentage of the game that was completed.