I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing

TL;DR

I2E reframes text-guided image editing as an interactive process within a structured environment, moving away from end-to-end pixel inpainting. The Decomposer converts images into manipulable object layers with explicit spatial relations, while the VLA Editor uses physics-aware chain-of-thought reasoning to produce a sequence of atomic, local actions. This decomposition enables stable, multi-round edits with high physical plausibility and precise localization, demonstrated by a new I2E-Bench and strong results on MagicBrush and EmuEdit. The approach reduces error accumulation across rounds and improves instruction compliance, compositionality, and spatial control, offering a scalable path toward more reliable and interpretable image editing.

Abstract

Existing text-guided image editing methods primarily rely on end-to-end pixel-level inpainting paradigm. Despite its success in simple scenarios, this paradigm still significantly struggles with compositional editing tasks that require precise local control and complex multi-object spatial reasoning. This paradigm is severely limited by 1) the implicit coupling of planning and execution, 2) the lack of object-level control granularity, and 3) the reliance on unstructured, pixel-centric modeling. To address these limitations, we propose I2E, a novel "Decompose-then-Action" paradigm that revisits image editing as an actionable interaction process within a structured environment. I2E utilizes a Decomposer to transform unstructured images into discrete, manipulable object layers and then introduces a physics-aware Vision-Language-Action Agent to parse complex instructions into a series of atomic actions via Chain-of-Thought reasoning. Further, we also construct I2E-Bench, a benchmark designed for multi-instance spatial reasoning and high-precision editing. Experimental results on I2E-Bench and multiple public benchmarks demonstrate that I2E significantly outperforms state-of-the-art methods in handling complex compositional instructions, maintaining physical plausibility, and ensuring multi-turn editing stability.

Figures (13)

• Figure 1: Paradigm Comparison. Unlike the Pixel Redrawing Paradigm that directly manipulates pixels, I2E transforms images into a structured environment, enabling the VLA Editor to perform spatial and physical reasoning for precise, physically plausible edits.
• Figure 2: Overview of the I2E. The Decomposer transforms unstructured images into a structured environment of actionable physical layers. The physics-aware VLA Editor then uses chain-of-thought reasoning to translate instructions into executable atomic actions (see bottom) and executes them sequentially.
• Figure 3: Multi-turn Stability.Top: Baselines exhibit severe error accumulation (e.g., visual distortion) over 4 rounds, while I2E preserves integrity. Bottom: Metrics confirm I2E's constant consistency in Saturation and Pixel Difference (PixDiff) versus the monotonic degradation of end-to-end models.
• Figure 4: The results of the qualitative comparison on I2E-Bench.
• Figure 5: Qualitative results on I2E-Bench compared to commercial models.

Theorems & Definitions (5)

• Theorem A.1: Instruction Collapse under Global Conditioning
• proof : Proof Sketch
• Proposition A.2: Elimination of Same-Step Gradient Conflict by Sequential Decomposition
• proof : Proof Sketch
• proof : Explanation