MSRAMIE: Multimodal Structured Reasoning Agent for Multi-instruction Image Editing

Abstract

Existing instruction-based image editing models perform well with simple, single-step instructions but degrade in realistic scenarios that involve multiple, lengthy, and interdependent directives. A main cause is the scarcity of training data with complex multi-instruction annotations. However, it is costly to collect such data and retrain these models. To address this challenge, we propose MSRAMIE, a training-free agent framework built on Multimodal Large Language Model (MLLM). MSRAMIE takes existing editing models as plug-in components and handle multi-instruction tasks via structured multimodal reasoning. It orchestrates iterative interactions between an MLLM-based Instructor and an image editing Actor, introducing a novel reasoning topology that comprises the proposed Tree-of-States and Graph-of-References. During inference, complex instructions are decomposed into multiple editing steps which enable state transitions, cross-step information aggregation, and original input recall, which enables systematic exploration of the image editing space and flexible progressive output refinement. The visualizable inference topology further provides interpretable and controllable decision pathways. Experiments show that as the instruction complexity increases, MSRAMIE can improve instruction following over 15% and increases the probability of finishing all modifications in a single run over 100%, while preserving perceptual quality and maintaining visual consistency.

Figures (11)

• Figure 1: Performance Degradation Under Multi-instruction. Existing models degrade when directly dealing with lengthy multi-instructions, manifested by poor instruction following and identity preservation that worsen with increasing complexity (1st row). MSRAMIE mitigates these issues (2nd row). Colored text under the input image stands for different parts of the editing multi-instructions, corresponding to color blocks under each image. A red cross stands for a failed instruction.
• Figure 2: MSRAMIE Architecture. Iterative interactions between the Instructor and Actor progressively construct an inference topology, where each state corresponds to one round of interaction. Transition and reference links among states form a Tree-of-States and a Graph-of-References, enabling multiple decision paths, or Chain-of-States, containing progressively refined candidate solutions in the image editing space. All states retain access to the root state for the original inputs during their creation.
• Figure 3: Image Editing Space Exploration by Structured Multimodal Reasoning. MSRAMIE creates a visualizable reasoning topology. States on a decision path dynamically focus on different parts of the original lengthy multi-instruction, with referencing to nearby states to prevent duplicated exploration. Inference topology expands in depth-first style, providing a transparent and controllable editing process.
• Figure 4: Instructor-Actor Interaction Process (left) and Instructor Submodules Process (right). Four Instructor submodules coordinate with the Actor to create a new state. One may refer to \ref{['fig:Fig2']} for the meaning of state colors.
• Figure 5: A Sample of Complex-Edit. Each non-synthetic image is accompanied with lengthy multi-instructions of 7 complexity levels. Higher complexity leads to more and lengthier editing requirements.