VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents

TL;DR

VisGym introduces a diverse, controllable suite of 17 long-horizon visually interactive environments to diagnose and train vision–language models on multi-step tasks. It integrates function-conditioned actions, textual feedback, and solver-generated demonstrations to enable systematic perturbations and supervised finetuning. Across 12 frontier models, significant gaps emerge in long-horizon decision-making, context usage, and grounding, but targeted ablations reveal concrete pathways—such as information-revealing demonstrations and joint vision–LLM finetuning—that yield substantial gains. The work provides a scalable platform for cross-domain analysis and a blueprint for improving multimodal agents in visually rich, interactive settings.

Abstract

Modern Vision-Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGym, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: https://visgym.github.io/.

Figures (33)

• Figure 1: An overview of VisGym. (Left) VisGym consists of $17$ diverse, long-horizon environments designed to systematically evaluate, diagnose, and train VLMs on visually interactive tasks with different domains, levels of state observability, and types of observations. (Right) An example trajectory for the Maze 3D navigation task illustrates a partially observable environment consisting of non-structured synthetic renderings. Here, a VLM is prompted with (1) the task description (simplified in the figure) and (2) a set of available actions to use (not shown in the figure for simplicity). The agent must select each action conditioned on both its past actions and observation history for its decision-making.
• Figure 2: Average task success rate for frontier models and our finetuned models. Proprietary models are in bold and our finetuned models are italicized.
• Figure 3: Density curve of steps taken for successful trajectories. Colored dashed line marks each model’s mean number of steps.
• Figure 4: Task success rate of frontier and finetuned models. Proprietary models are shown in bold, and our finetuned models in italics. (E) and (H) denote easy and hard task settings. Darker cells indicate higher success rates. Models are ordered by average task performance (top = better), and tasks by average model performance, excluding our finetuned ones (right = harder).
• Figure 5: Effect of truncating conversational context on model performance. The settings $1$, $2$, $4$, and $\infty$ correspond to retaining only the current turn, the current $+$ previous turn, the current $+$ previous $3$ turns, and the full history, respectively. Error bars show the standard error of the mean.