Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin

TL;DR

This work tackles the gap between geometry and texture in 3D reconstruction for Digital Twins by proposing a camera-only pipeline that uses Gaussian Splatting for photorealistic surfaces and explicit meshes for physics-based simulation. It introduces monocular material extraction, per-pixel projection of 2D materials onto 3D Gaussians, and a material-to-BSDF mapping via Matsynth, enabling realistic LiDAR reflectivity in simulated sensors. Through validation on an instrumented vehicle dataset, the approach achieves sensor-simulation accuracy comparable to LiDAR-camera fusion while avoiding hardware calibration, demonstrating practical utility for ADAS development and Digital Twin workflows. The results highlight the trade-offs between pixel-level fidelity and perceptual quality, with ablations confirming the value of shape-aware segmentation for stable material labeling. Overall, the pipeline provides a modular, camera-only alternative that preserves photorealism and material fidelity for physics-based sensor simulation.

Abstract

3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.

Figures (5)

• Figure 1: Overview of our camera-only reconstruction pipeline. From RGB images, we: (a) extract semantic material masks, (b) reconstruct the 3D scene, (c) project material labels onto mesh surfaces, (d) assign physics-based materials, and (e) validate through sensor simulation.
• Figure 2: Shape-aware material refinement. From left to right: input RGB image, RMSNet predictions, FastSAM object boundaries, and final result after majority voting. The combination produces consistent material labels with sharp edges. Example from our internal dataset.
• Figure 3: Qualitative comparison of novel view rendering. From left to right: ground truth RGB, our adapted H3DGS model, 3DGS, CityGaussianV2, and 3DGUT. Our model achieves competitive visual quality comparable to state-of-the-art baselines.
• Figure 4: From left to right: real-world LiDAR ground truth from our instrumented vehicle, LiDAR-camera fusion baseline simulated in Prescan, and our camera-only reconstruction simulated in Prescan.
• Figure 5: Distribution of absolute reflectivity errors for camera-only (ours) and LiDAR-based reconstruction methods. Our camera-only approach achieves comparable median error with slightly higher variability.