Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing

TL;DR

RISEBench introduces the first dedicated benchmark for Reasoning-Informed Visual Editing, evaluating temporal, causal, spatial, and logical reasoning in image editing. The framework uses three evaluation dimensions—Instruction Reasoning, Appearance Consistency, and Visual Plausibility—assessed by humans and a robust LMM-as-a-judge, with strong correlation between judgments. Experiments across eight prominent models reveal substantial gaps in reasoning-based editing, with GPT-4o-Image achieving the best performance (about 28.9%), yet logical reasoning remains a major bottleneck. The work provides a structured methodology, revealing practical limitations and guiding future research, and releases code and data for community use.

Abstract

Large Multi-modality Models (LMMs) have made significant progress in visual understanding and generation, but they still face challenges in General Visual Editing, particularly in following complex instructions, preserving appearance consistency, and supporting flexible input formats. To study this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE). RISEBench focuses on four key reasoning categories: Temporal, Causal, Spatial, and Logical Reasoning. We curate high-quality test cases for each category and propose an robust evaluation framework that assesses Instruction Reasoning, Appearance Consistency, and Visual Plausibility with both human judges and the LMM-as-a-judge approach. We conducted experiments evaluating nine prominent visual editing models, comprising both open-source and proprietary models. The evaluation results demonstrate that current models face significant challenges in reasoning-based editing tasks. Even the most powerful model evaluated, GPT-4o-Image, achieves an accuracy of merely 28.8%. RISEBench effectively highlights the limitations of contemporary editing models, provides valuable insights, and indicates potential future directions for the field of reasoning-aware visual editing. Our code and data have been released at https://github.com/PhoenixZ810/RISEBench.

Figures (25)

• Figure 1: Examples of leading models on the Reasoning-Informed viSual Editing(RISE) benchmark. RISEBench contains complex and various tasks that pose challenges to current models.
• Figure 2: Overview of RISEBench. We present illustrative example questions from each of the four problem categories, each demanding profound image understanding and reasoning capabilities.
• Figure 3: Task Distribution of RISEBench. RISEBench contains four main reasoning categories: Temporal, Causal, Spatial, and Logical. Each category includes various subtasks, facilitating a comprehensive evaluation.
• Figure 4: Evaluation metrics of RISEBench. RISEBench assesses the quality of generated images along three key dimensions: Instruction Following, Appearance Consistency, and Visual Plausibility. For each dimension, carefully crafted prompts are provided to the evaluator model (GPT-4.1 in this study), which analyzes various inputs and returns scores for each corresponding sub-dimension.
• Figure 5: Comparison across models on three evaluation sub-dimensions. GPT-4o-Image demonstrates superior performance, achieving the highest scores across all three evaluation metrics. Gemini-2-Flash-Series also exhibits competitive performance on these criteria. In contrast, the performance of many other evaluated models was considerably lower, indicating significant limitations in their ability to follow instructions and maintain visual integrity.