4DGS-Craft: Consistent and Interactive 4D Gaussian Splatting Editing

TL;DR

4DGS-Craft tackles inconsistent editing in dynamic 4D Gaussian Splatting by integrating a 4D-aware supervision pipeline with a multi-view grid, a Gaussian selection mechanism, and an LLM-based intent understanding module. It introduces a 4D-aware InstructPix2Pix that leverages 4D geometry features (VGGT) and multi-view inputs to enforce view and temporal consistency, while a Gaussian selector preserves non-edited regions during optimization. An LLM-driven intent module decomposes open-ended user prompts into atomic editing tasks, enabling complex and abstract instructions to be executed as sequences of concrete operations. Through experiments on DyCheck and N3DV, 4DGS-Craft outperforms baselines in both qualitative and quantitative measures of consistency and instruction adherence, enabling more robust and intuitive 4D scene editing for practical applications.

Abstract

Recent advances in 4D Gaussian Splatting (4DGS) editing still face challenges with view, temporal, and non-editing region consistency, as well as with handling complex text instructions. To address these issues, we propose 4DGS-Craft, a consistent and interactive 4DGS editing framework. We first introduce a 4D-aware InstructPix2Pix model to ensure both view and temporal consistency. This model incorporates 4D VGGT geometry features extracted from the initial scene, enabling it to capture underlying 4D geometric structures during editing. We further enhance this model with a multi-view grid module that enforces consistency by iteratively refining multi-view input images while jointly optimizing the underlying 4D scene. Furthermore, we preserve the consistency of non-edited regions through a novel Gaussian selection mechanism, which identifies and optimizes only the Gaussians within the edited regions. Beyond consistency, facilitating user interaction is also crucial for effective 4DGS editing. Therefore, we design an LLM-based module for user intent understanding. This module employs a user instruction template to define atomic editing operations and leverages an LLM for reasoning. As a result, our framework can interpret user intent and decompose complex instructions into a logical sequence of atomic operations, enabling it to handle intricate user commands and further enhance editing performance. Compared to related works, our approach enables more consistent and controllable 4D scene editing. Our code will be made available upon acceptance.

Figures (7)

• Figure 1: We present 4DGS-Craft, an intuitive and consistent 4D scene editing framework that accommodates a wide range of user instructions, including highly abstract and complex commands, and delivers visually pleasing results.
• Figure 2: Framework. (a) 4DGS-Craft Pipeline, which first employs the LLM-based Intent Understanding Module to decompose the editing prompt into atomic editing operations and subsequently applies the Gaussian Editor to edit the 4DGS based on those atomic tasks. (b) Details of the Gaussian Editor, which performs 4DGS editing in three stages: 4D-aware InstructPix2Pix optimization, Gaussian Selector optimization, and 4D Gaussian optimization.
• Figure 3: Qualitative Comparison. We provided the qualitative comparison with both the 4DGS-based editing method CTRL-D and the dynamic NeRF-based editing method Instruct 4D-to-4D. Our results demonstrate superior prompt-following capability and higher consistency across views and time.
• Figure 4: Gaussian Mask Tracking. We propose tracking Gaussian masks during the clone, split, and prune operations performed on the canonical 3D Gaussian. An indicator value of 0 denotes the non-editing region, while a value of 1 denotes the editing region. Gaussians assigned a value of 0 will not be optimized during editing.
• Figure 5: (a) Ablation study of Gaussian Selector on the "mochi-high-five" scene from the DyCheck dataset. Our complete method demonstrates superior preservation of texture details in the non-edited area. (b) Ablation study of the LLM-based Intent Understanding Module (LIUM). $\mathrm{Ours^*}$ denotes our method without the 4D-aware IP2P and the Gaussian Selector. LIUM empowers our model to produce outputs with greater clarity and finer details in the edited area.