Towards Realistic Example-based Modeling via 3D Gaussian Stitching

TL;DR

The paper tackles realistic example-based 3D modeling by stitching real-world parts represented as 3D Gaussian fields. It introduces Seamless Gaussians, combining sampling-based cloning and clustering-based tuning to harmonize texture and structure, with a GUI for real-time segmentation and transformation. The approach overcomes gradient-based and grid limitations of prior methods like SeamlessNeRF, demonstrated through qualitative Real-world results and VQA-based quantitative scores on real data. This work enables practical, interactive creation of photorealistic 3D composites from real scenes, with potential impact on editing and synthesis workflows.

Abstract

Using parts of existing models to rebuild new models, commonly termed as example-based modeling, is a classical methodology in the realm of computer graphics. Previous works mostly focus on shape composition, making them very hard to use for realistic composition of 3D objects captured from real-world scenes. This leads to combining multiple NeRFs into a single 3D scene to achieve seamless appearance blending. However, the current SeamlessNeRF method struggles to achieve interactive editing and harmonious stitching for real-world scenes due to its gradient-based strategy and grid-based representation. To this end, we present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis. Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models represented by 3D Gaussian Splatting (3DGS). For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported. Thus, a novel sampling-based cloning method is proposed to harmonize the blending while preserving the original rich texture and content. Our workflow consists of three steps: 1) real-time segmentation and transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis to identify boundary points in the intersecting area between the source and target models, and 3) two-phase optimization of the target model using sampling-based cloning and gradient constraints. Extensive experimental results validate that our approach significantly outperforms previous works in terms of realistic synthesis, demonstrating its practicality. More demos are available at https://ingra14m.github.io/gs_stitching_website.

Figures (16)

• Figure 1: Overview of our framework. Our novel pipeline provides an interactive editing experience and has real-time previewing capabilities to visualize the optimizing process, allowing for the seamless and interactive combination of multiple Gaussian fields.
• Figure 2: For a Gaussian point in the target field, its (a) K-nearest neighbors in the source field can be leveraged to justify whether this point belongs to the intersection boundary region. We use the boundary of (b) as an example to demonstrate the effectiveness of this strategy, as shown in (c).
• Figure 3: ablation study on the color loss in the S-phase. Without color loss, the propagation is inefficient and will not begin. The cases shown above have been running for more than twice as long, but they are still trapped in insufficient propagation. It is because, without color loss, only a small number of points' features need to be updated at first, as opposed to shared weights in an MLP applied to all points. That minor "forces" cannot drive the overall minimization of the gradient loss.
• Figure 4: ablation study on the effectiveness of gradient loss for different weights. Experiments show that higher weights can help to preserve more content while preventing harmonization.
• Figure 5: ablation study on sampling-based cloning (S.) and clustering-based tuning (T.). Here, "Both" means the full scheme.