ProGDF: Progressive Gaussian Differential Field for Controllable and Flexible 3D Editing

TL;DR

The paper addresses the slow, iterative workflow of 3D Gaussians editing by introducing ProGDF, an out-of-loop framework that enables real-time, controllable 3D editing from a single training session. It combines Progressive Gaussian Splatting, which yields diverse high-quality intermediate editing results, with Gaussian Differential Field, a lightweight MLP that maps edits to per-Gaussian offsets and supports a user-controlled slider for variable outcomes. The method achieves superior user satisfaction and CLIP directional similarity compared to baselines and enables flexible fine-grained manipulation by region-specific editing. Practically, ProGDF significantly improves the user experience in 3D editing by delivering fast, controllable, and high-quality edits suitable for asset editing and reuse in VR, gaming, and film pipelines.

Abstract

3D editing plays a crucial role in editing and reusing existing 3D assets, thereby enhancing productivity. Recently, 3DGS-based methods have gained increasing attention due to their efficient rendering and flexibility. However, achieving desired 3D editing results often requires multiple adjustments in an iterative loop, resulting in tens of minutes of training time cost for each attempt and a cumbersome trial-and-error cycle for users. This in-the-loop training paradigm results in a poor user experience. To address this issue, we introduce the concept of process-oriented modelling for 3D editing and propose the Progressive Gaussian Differential Field (ProGDF), an out-of-loop training approach that requires only a single training session to provide users with controllable editing capability and variable editing results through a user-friendly interface in real-time. ProGDF consists of two key components: Progressive Gaussian Splatting (PGS) and Gaussian Differential Field (GDF). PGS introduces the progressive constraint to extract the diverse intermediate results of the editing process and employs rendering quality regularization to improve the quality of these results. Based on these intermediate results, GDF leverages a lightweight neural network to model the editing process. Extensive results on two novel applications, namely controllable 3D editing and flexible fine-grained 3D manipulation, demonstrate the effectiveness, practicality and flexibility of the proposed ProGDF.

Figures (10)

• Figure 1: (a): Existing in-the-loop training for 3D Gaussians editing, takes tens of minutes for each loop, resulting in a cumbersome trial-and-error cycle for users. (b): Our out-of-loop training approach, requiring only a single training session to provide controllable editing capability and variable editing results through a user-friendly interface, with each adjustment taking only $0.02$ seconds.
• Figure 2: Results of ProGDF. Our ProGDF not only achieves controllable 3D editing with a user-friendly interface to generate diverse results at varying levels of editing (see first row), but also enables flexible fine-grained 3D manipulation to create new results from existing edited 3D scenes (see second row). Note that the instructions marked in the second row are not real inputs, but are used to indicate which editing results they are combined from.
• Figure 3: Overview of ProGDF. ProGDF contains two key components: Progressive Gaussian Splatting (PGS) and Gaussian Differential Field (GDF). PGS is utilized to extract diverse high-quality intermediate results by tracking the optimization trajectory of 3D Gaussians from the original scene to the target scene. GDF leverages a lightweight MLP to model the 3D editing process as an estimation of the difference between the original and the edited 3D Gaussians, providing controllable editing capability and variable editing results through a user-friendly interface in real-time.
• Figure 4: Results of controllable 3D editing. Our method is capable of editing a variety of scenes. Only the 3D Gaussians within the region to be edited are inputted into the GDF, leaving the other region unaffected. We present two editing instructions for each scene, with two different results for each instruction, to demonstrate the controllability of the editing.
• Figure 5: Comparison with previous methods. We compare our method with Instruct-N2N haque2023instruct (NeRF-based) and GaussianEditor chen2024gaussianeditor (3DGS-based), respectively. Neither allows for controllable editing, whereas our method is capable of controlling the editing result through a user-friendly interface.