Enhancement of 3D Gaussian Splatting using Raw Mesh for Photorealistic Recreation of Architectures

TL;DR

This paper addresses enhancing photorealistic reconstruction of architectural scenes by augmenting 3D Gaussian Splatting with raw mesh priors to better capture shape and texture under non-systematic image capture. The method samples points on coarse raw meshes using barycentric coordinates, assigns per-triangle sampling density by area, initializes colors via K-means clustering of image colors, and aligns the mesh-derived point cloud with COLMAP, feeding 3DGS. Evaluated on four drone-captured scenes, the approach yields notable improvements in geometry and texture over vanilla 3DGS, as quantified by SSIM and PSNR. The results demonstrate a practical route for architectural visualization by integrating publicly available raw models with fast radiance-field rendering.

Abstract

The photorealistic reconstruction and rendering of architectural scenes have extensive applications in industries such as film, games, and transportation. It also plays an important role in urban planning, architectural design, and the city's promotion, especially in protecting historical and cultural relics. The 3D Gaussian Splatting, due to better performance over NeRF, has become a mainstream technology in 3D reconstruction. Its only input is a set of images but it relies heavily on geometric parameters computed by the SfM process. At the same time, there is an existing abundance of raw 3D models, that could inform the structural perception of certain buildings but cannot be applied. In this paper, we propose a straightforward method to harness these raw 3D models to guide 3D Gaussians in capturing the basic shape of the building and improve the visual quality of textures and details when photos are captured non-systematically. This exploration opens up new possibilities for improving the effectiveness of 3D reconstruction techniques in the field of architectural design.

Figures (10)

• Figure 1: A comparison between vanilla 3DGS and our raw model enhanced method. The centers of the Gaussians after training are demonstrated in the first row. Vanilla 3DGS fails to represent the basic geometry and texture of the wall due to inadequate observation, but ours succeeds. The rendered scene is shown in the second row. Please note the windows in the red box, which denote our enhanced performance in capturing realistic details.
• Figure 2: Example of raw 3D models from Cesium OSM Buildings (figure from Cesium website). The raw models already demonstrate the basic shape of the buildings, but obviously without detailed appearance.
• Figure 3: The pipeline of our method.
• Figure 4: Demonstration of raw mesh models and point clouds sampled with our method of each scene in Blender.
• Figure 5: Comparison between the raw model with no detail and the scene rendered with our output. It is obvious that the mesh provides no detailed structures, and they must be reconstructed during training of the 3DGS.