Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Multi-View Reconstruction via Texture-Guided Gaussian-Mesh Joint Optimization

Zhejia Cai, Puhua Jiang, Shiwei Mao, Hongkun Cao, Ruqi Huang

TL;DR

This work introduces a unified framework for joint geometry and appearance optimization in multi-view reconstruction by integrating Gaussian-mesh representations with differentiable rendering. Starting from a coarse textured mesh obtained via 3D Gaussian Splatting, it performs texture-guided remeshing with edge-length control and inverse-rendering–driven refinement, producing high-fidelity, editable meshes. A vertex-Gaussian binding scheme then transfers the improved geometry to Gaussians, enabling cohesive relighting and deformation editing. Experiments on DTU and DTC datasets show improvements in geometry and rendering quality, as well as improved downstream editing performance, with ablations highlighting the importance of RGB supervision and texture-aware remeshing. The approach offers a practical path toward editable, high-quality textured reconstructions for AR/VR and digital content creation.

Abstract

Reconstructing real-world objects from multi-view images is essential for applications in 3D editing, AR/VR, and digital content creation. Existing methods typically prioritize either geometric accuracy (Multi-View Stereo) or photorealistic rendering (Novel View Synthesis), often decoupling geometry and appearance optimization, which hinders downstream editing tasks. This paper advocates an unified treatment on geometry and appearance optimization for seamless Gaussian-mesh joint optimization. More specifically, we propose a novel framework that simultaneously optimizes mesh geometry (vertex positions and faces) and vertex colors via Gaussian-guided mesh differentiable rendering, leveraging photometric consistency from input images and geometric regularization from normal and depth maps. The obtained high-quality 3D reconstruction can be further exploit in down-stream editing tasks, such as relighting and shape deformation. The code will be publicly available upon acceptance.

Improving Multi-View Reconstruction via Texture-Guided Gaussian-Mesh Joint Optimization

TL;DR

This work introduces a unified framework for joint geometry and appearance optimization in multi-view reconstruction by integrating Gaussian-mesh representations with differentiable rendering. Starting from a coarse textured mesh obtained via 3D Gaussian Splatting, it performs texture-guided remeshing with edge-length control and inverse-rendering–driven refinement, producing high-fidelity, editable meshes. A vertex-Gaussian binding scheme then transfers the improved geometry to Gaussians, enabling cohesive relighting and deformation editing. Experiments on DTU and DTC datasets show improvements in geometry and rendering quality, as well as improved downstream editing performance, with ablations highlighting the importance of RGB supervision and texture-aware remeshing. The approach offers a practical path toward editable, high-quality textured reconstructions for AR/VR and digital content creation.

Abstract

Reconstructing real-world objects from multi-view images is essential for applications in 3D editing, AR/VR, and digital content creation. Existing methods typically prioritize either geometric accuracy (Multi-View Stereo) or photorealistic rendering (Novel View Synthesis), often decoupling geometry and appearance optimization, which hinders downstream editing tasks. This paper advocates an unified treatment on geometry and appearance optimization for seamless Gaussian-mesh joint optimization. More specifically, we propose a novel framework that simultaneously optimizes mesh geometry (vertex positions and faces) and vertex colors via Gaussian-guided mesh differentiable rendering, leveraging photometric consistency from input images and geometric regularization from normal and depth maps. The obtained high-quality 3D reconstruction can be further exploit in down-stream editing tasks, such as relighting and shape deformation. The code will be publicly available upon acceptance.

Paper Structure

This paper contains 21 sections, 14 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Surface Reconstruction with Volume Rendering
  4. Hybrid of Gaussian Splatting and Mesh Representations
  5. Reconstruction via Optimization
  6. Method
  7. Geometry-Color Remeshing Operations
  8. Texture-Based Edge Length Control
  9. Mesh Optimization via Inverse Rendering
  10. Vertex-Gaussian Binding for Relighting and Deformation
  11. Experiments
  12. Remeshing evaluation
  13. Relighting and Deformation Editing
  14. Ablation Study
  15. Limitations and Conclusion
  16. ...and 6 more sections

Figures (7)

  • Figure 1: The schematic illustration of our pipeline.
  • Figure 2: Remeshing results with (middle) and without (right) texture density based edge length control (i.e., TELC).
  • Figure 3: Qualitative results on DTU and DTC dataset
  • Figure 4: Qualitative results on Synthetic4Relight dataset
  • Figure 5: Comparison of points distribution between Ours and R3DG
  • ...and 2 more figures