How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing

TL;DR

VIBE introduces a cognitively motivated benchmark for visual instruction–driven image editing, organizing tasks into a three-level hierarchy—deictic grounding, morphological manipulation, and causal reasoning—across 1,034 samples and 10 subtasks. It couples this with an LMM-as-a-judge evaluation framework to assess 17 open-source and proprietary models, revealing early-stage visual instruction capabilities in proprietary systems, a wide gap to open-source alternatives, and systematic performance degradation as task complexity increases. The study demonstrates strong alignment between LMM-based evaluations and human judgments, analyzes style- and multi-task–dependent behavior, and highlights the complementary roles of textual and visual instructions in grounding and semantic specification. Overall, VIBE provides a scalable, nuanced platform to drive advances in multimodal instruction-following for image editing and to guide future research in grounding, reasoning, and cross-modal coordination.

Abstract

Recent generative models have achieved remarkable progress in image editing. However, existing systems and benchmarks remain largely text-guided. In contrast, human communication is inherently multimodal, where visual instructions such as sketches efficiently convey spatial and structural intent. To address this gap, we introduce VIBE, the Visual Instruction Benchmark for Image Editing with a three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning. Across these levels, we curate high-quality and diverse test cases that reflect progressively increasing complexity in visual instruction following. We further propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment. Through a comprehensive evaluation of 17 representative open-source and proprietary image editing models, we find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models. However, performance degrades markedly with increasing task difficulty even for the strongest systems, highlighting promising directions for future research.

Figures (13)

• Figure 1: Motivation and scope of the VIBE benchmark. Traditional image editing is largely text-guided, where conveying spatial intent relies on verbose descriptions and incurs high cognitive load. In contrast, visual instructions enable precise and explicit grounding, providing a more human-aligned interaction paradigm. VIBE is designed to fill the evaluation gap by systematically benchmarking this visual intruction-guided multi-modal image editing.
• Figure 2: Composition of VIBE. VIBE comprises 1,034 samples across 10 tasks, organized into a three-level hierarchy that reflects increasing interaction and reasoning complexity, from deictic grounding and morphological manipulation to causal reasoning.
• Figure 3: Overview of VIBE. VIBE organizes visual instruction-guided image editing into a three-level interaction hierarchy with increasing task complexity. The Deictic Level treats visual instructions as selectors that specify localized regions or objects for basic spatial operations. The Morphological Level interprets visual instructions as blueprints that define abstract structural constraints. The Causal Level views visual instructions as catalysts that encode underlying physical or logical dynamics.
• Figure 4: Performance across image styles on the Deictic Level. Left: Average Deictic Level scores across real-world, animation, and sketch images for four proprietary models. Right: Metric-level heatmaps for Seedream 4.5 and GPT-Image-1, illustrating style-dependent variations in Instruction Adherence, Contextual Preservation, and Visual Coherence.
• Figure 5: Style-wise performance on Draft Instantiation.