TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts

TL;DR

TIP-Editor tackles the challenge of precise localized 3D scene editing guided by both text and image prompts. It introduces a three-stage pipeline that combines stepwise 2D personalization with an explicit 3D Gaussian Splatting representation, using SDS-based coarse editing and pixel-level refinement to align edits with both the textual and visual references. Key contributions include an attention-guided localization mechanism, LoRA-based content personalization for reference images, and the use of GS to enable accurate, background-preserving local edits. Experiments across diverse scenes show substantial gains in editing quality and prompt alignment over state-of-the-art baselines, highlighting the method's practical impact for controllable 3D editing.

Abstract

Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIPEditor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.

Figures (11)

• Figure 1: Method overview. TIP-Editor optimizes a 3D scene that is represented as 3D Gaussian splatting (GS) to conform with a given hybrid text-image prompt. The editing process includes three stages: 1) a stepwise 2D personalization strategy, which features a localization loss in the scene personalization step and a separate novel content personalization step dedicated to the reference image based on LoRA (Sec. \ref{['subsec:learning']}); 2) a coarse editing stage using SDS (Sec. \ref{['subsec:coarse']}); and 3) a pixel-level texture refinement stage, utilizing carefully generated pseudo-GT image from both the rendered image $I_c$ and the denoised image $I_c^{d}$ (Sec. \ref{['subsec:refinement']}).
• Figure 2: Editing results of the proposed TIP-Editor. Images in the text prompts denote their associated rare tokens, which are fixed without optimization.
• Figure 3: Sequential editing results. We show two rendered images of the 3D scene after every editing step, indicated by the number in the top-left corner. $V_*$, $V_{**}$, and $V_{***}$ represent the special tokens of the scene in different sequences of editing.
• Figure 4: Results of using a generated image as the reference. We first generate several candidate images by the diffusion model using text prompts, then we choose one as the reference image for editing.
• Figure 5: Visual comparisons between different methods. Our method produces obviously higher-quality results and accurately follows the reference image input (bottom-right corner in column 1). Instruct-N2N sometimes misunderstands (row 1) or overlooks (row 2) the keywords. DreamEditor faces difficulty in making obvious shape changes (row 2). Both of them do not support image prompts to specify detailed appearance/style, producing less controlled results.