ICE-G: Image Conditional Editing of 3D Gaussian Splats

TL;DR

ICE-G presents a fast, region-aware approach to editing colors and textures in 3D scenes from a single reference image or view. By combining SAM-based segmentation, DINO-based cross-view region matching, Texture Reformer for texture transfer, and NNFM plus standard image losses, ICE-G achieves per-region style transfers that are consistent across multiple views and compatible with Gaussian Splats and NeRFs. The method delivers higher-quality edits with fine-grained control, demonstrated on multiple datasets and supported by a user study favoring ICE-G over strong baselines. This work enables creative, view-consistent 3D appearance edits with practical speed advantages, while acknowledging current limitations in segmentation granularity and reflective texture preservation.

Abstract

Recently many techniques have emerged to create high quality 3D assets and scenes. When it comes to editing of these objects, however, existing approaches are either slow, compromise on quality, or do not provide enough customization. We introduce a novel approach to quickly edit a 3D model from a single reference view. Our technique first segments the edit image, and then matches semantically corresponding regions across chosen segmented dataset views using DINO features. A color or texture change from a particular region of the edit image can then be applied to other views automatically in a semantically sensible manner. These edited views act as an updated dataset to further train and re-style the 3D scene. The end-result is therefore an edited 3D model. Our framework enables a wide variety of editing tasks such as manual local edits, correspondence based style transfer from any example image, and a combination of different styles from multiple example images. We use Gaussian Splats as our primary 3D representation due to their speed and ease of local editing, but our technique works for other methods such as NeRFs as well. We show through multiple examples that our method produces higher quality results while offering fine-grained control of editing. Project page: ice-gaussian.github.io

Figures (14)

• Figure 1: Our method, ICE-G, allows for quick color or texture edits to a 3D scene given a single style image, or mask selection on a single view. We show two rendered views of mask select editing for the Garden Scene where we apply stone texture to the table and fall colors to the grass (left). We also show two renders of correspondance based editing where we can transfer the color of the blue car to the lego and the texture of the grass to the table (right).
• Figure 2: The user supplied style image is segmented and its masked regions are matched with masked regions of sampled datset views via DINO correspondences. The color/texture is then transferred to those matching regions, and the splat is edited with this updated dataset.
• Figure 3: Comparing different dataset sampling rates for turning the road to a river. Sampling 5% or 10% of images from a dataset to edit results in numerous artifacts and other degradations, and quality peaks at around 20% sampling.
• Figure 4: Adding texture to the garden table from mip-NeRF360 (left). Using paintings to texture the table (right).
• Figure 5: Comparison of our method in local texture editing of ours and baselines.