Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine

TL;DR

FFSE introduces a 3D-aware autoregressive framework for multi-round object manipulation directly on real images, modeling edits as sequences of $3D$ transformations and maintaining global scene consistency. It relies on a novel hybrid dataset, 3DObjectEditor, combining real and synthetic sequences to learn robust object/background interactions, including shadows and occlusions, across rounds. The architecture enhances history integration via frame and operation encoders, operation self-attention, and context self-attention, built atop a diffusion backbone with a two-stage training regime using Domain LoRA to bridge domains. Empirical results show FFSE surpasses state-of-the-art image-space and 3D-space methods in single-round and multi-round editing, validated by quantitative metrics and human studies, with strong generalization to real-world scenes. This work offers a practical, scalable approach to 3D-aware image editing suitable for iterative workflows without heavy per-edit 3D reconstruction.

Abstract

Recent advances in text-to-image (T2I) diffusion models have significantly improved semantic image editing, yet most methods fall short in performing 3D-aware object manipulation. In this work, we present FFSE, a 3D-aware autoregressive framework designed to enable intuitive, physically-consistent object editing directly on real-world images. Unlike previous approaches that either operate in image space or require slow and error-prone 3D reconstruction, FFSE models editing as a sequence of learned 3D transformations, allowing users to perform arbitrary manipulations, such as translation, scaling, and rotation, while preserving realistic background effects (e.g., shadows, reflections) and maintaining global scene consistency across multiple editing rounds. To support learning of multi-round 3D-aware object manipulation, we introduce 3DObjectEditor, a hybrid dataset constructed from simulated editing sequences across diverse objects and scenes, enabling effective training under multi-round and dynamic conditions. Extensive experiments show that the proposed FFSE significantly outperforms existing methods in both single-round and multi-round 3D-aware editing scenarios.

Figures (16)

• Figure 1: 3D-aware object manipulation results of our Free-Form Scene Editor (FFSE). 1) Object effects. FFSE can process a variety of 3D operations, including challenging transformations such as rotations. 2) Background effects. FFSE generates realistic environmental interaction resulting from object manipulations, such as shadows and occlusions. 3) Multi-round editing. FFSE maintains consistency of scene elements across multiple editing iterations. Moreover, the proposed FFSE provides a user-friendly interface without time-consuming 3D reconstruction.
• Figure 2: Overall framework of Free-Form Scene Editor (FFSE) with dashed boxes indicating introduced learnable modules, where the middle blocks and convolutional layers from the base model are omitted for simplicity. $N_d$ and $N_u$ denote the number of down and up blocks, respectively. Two 6-length editing sequences are shown as an example, where only $DL_\text{syn}$ is active since the current training batch is sampled from $D_\text{syn}$. Historical observations and operations are processed by frame encoder and operation encoder, respectively, to capture scene structure changes. The output of frame encoder is added to down block features, while the output from operation encoder is injected into the main branch via operation self-attention. Additionally, standard self-attention modules are enhanced by context self-attention to improve the appearance consistency of the edited object.
• Figure 3: Evaluation of object effects under different 3D operations in single-round editing.
• Figure 4: Evaluation of background effects in single-round editing. The figures demonstrate that FFSE generates more physically-plausible environmental interactions.
• Figure 5: Evaluation in multi-round editing. FFSE accomplishes the operation in each editing round, and maintains high consistency of scene elements. As indicated in columns 2-4 of our method, the cup is first occluded by the teapot, and then becomes visible.