OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation

TL;DR

OMEGAS tackles the problem of reconstructing high-fidelity meshes for a specified object within large open-world scenes, where occlusions and view gaps degrade traditional scene-wide reconstructions. It introduces Target Gaussian Segmentation via 2D Gaussian Splatting to obtain 3D-consistent target masks, and Target Replenishment that personalizes diffusion priors (e.g., Stable Diffusion) for inpainting unseen regions, followed by TSDF fusion to produce the final mesh. The approach yields significant improvements over prior methods in both mesh detail and occlusion robustness across multiple datasets, validating its effectiveness for object-level reconstruction in open-world contexts. These capabilities enable precise, controllable object meshes for applications in VR, rendering, and robotics where targeted reconstruction is essential.

Abstract

Recent advancements in 3D reconstruction technologies have paved the way for high-quality and real-time rendering of complex 3D scenes. Despite these achievements, a notable challenge persists: it is difficult to precisely reconstruct specific objects from large scenes. Current scene reconstruction techniques frequently result in the loss of object detail textures and are unable to reconstruct object portions that are occluded or unseen in views. To address this challenge, we delve into the meticulous 3D reconstruction of specific objects within large scenes and propose a framework termed OMEGAS: Object Mesh Extraction from Large Scenes Guided by Gaussian Segmentation. Specifically, we proposed a novel 3D target segmentation technique based on 2D Gaussian Splatting, which segments 3D consistent target masks in multi-view scene images and generates a preliminary target model. Moreover, to reconstruct the unseen portions of the target, we propose a novel target replenishment technique driven by large-scale generative diffusion priors. We demonstrate that our method can accurately reconstruct specific targets from large scenes, both quantitatively and qualitatively. Our experiments show that OMEGAS significantly outperforms existing reconstruction methods across various scenarios. Our project page is at: https://github.com/CrystalWlz/OMEGAS

Figures (7)

• Figure 1: OMEGAS segments and generates high-quality meshes for specified objects in open-world scenes (top). OMEGAS can effectively reconstruct the unseen portion of the target (bottom).
• Figure 2: The framework of OMEGAS. OMEGAS comprises two stages: Target Gaussian Segmentation to segment target 2DGS model from multi-view images and provide accurate 3D consistent masks; Target Replenishment to optimize the target model by personalized Stable Diffusion with a Mask-generation & Inpainting process. The final object mesh is extracted from the optimized target Gaussian model.
• Figure 3: Mask generation process.
• Figure 4: Results of meshes with OMEGAS on the Instruct-NeRF2NeRF dataset(line 1), Tanks&Temples dataset(line 2), LERF dataset(line 3,4,5) and Mip-Nerf360 dataset(line 6).
• Figure 5: Comparing mesh extracting results with SuGaR, 2D Gaussian Splatting and DreamGaussain. The scenes are selected from LERF and Instruct-NeRF2NeRF datasets. Both SuGaR, 2DGS, and our approach use multi-view images as input, while DreamGaussian relies on a single-view image.