Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing

Tian-Xing Xu, Wenbo Hu, Yu-Kun Lai, Ying Shan, Song-Hai Zhang

TL;DR

Texture-GS addresses the entanglement of geometry and appearance in 3D Gaussian Splatting by introducing a learnable 2D texture mapped onto the surface and a UV mapping MLP. It employs a Taylor-series-based approximation to map ray-Gaussian intersections to texture coordinates, enabling real-time rendering and flexible texture editing such as texture swapping and painting. The approach is trained in a two-stage process using surface-derived priors for UV mapping and a photometric/regularization loss for the texture, achieving high-fidelity texture reconstruction on the DTU dataset with about 58 FPS on consumer hardware. This disentangled representation enables practical appearance editing while preserving the efficiency and reconstruction quality of 3D-GS, representing a meaningful step toward editable, real-time 3D scene representations.

Abstract

3D Gaussian splatting, emerging as a groundbreaking approach, has drawn increasing attention for its capabilities of high-fidelity reconstruction and real-time rendering. However, it couples the appearance and geometry of the scene within the Gaussian attributes, which hinders the flexibility of editing operations, such as texture swapping. To address this issue, we propose a novel approach, namely Texture-GS, to disentangle the appearance from the geometry by representing it as a 2D texture mapped onto the 3D surface, thereby facilitating appearance editing. Technically, the disentanglement is achieved by our proposed texture mapping module, which consists of a UV mapping MLP to learn the UV coordinates for the 3D Gaussian centers, a local Taylor expansion of the MLP to efficiently approximate the UV coordinates for the ray-Gaussian intersections, and a learnable texture to capture the fine-grained appearance. Extensive experiments on the DTU dataset demonstrate that our method not only facilitates high-fidelity appearance editing but also achieves real-time rendering on consumer-level devices, e.g. a single RTX 2080 Ti GPU.

Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing

TL;DR

Texture-GS addresses the entanglement of geometry and appearance in 3D Gaussian Splatting by introducing a learnable 2D texture mapped onto the surface and a UV mapping MLP. It employs a Taylor-series-based approximation to map ray-Gaussian intersections to texture coordinates, enabling real-time rendering and flexible texture editing such as texture swapping and painting. The approach is trained in a two-stage process using surface-derived priors for UV mapping and a photometric/regularization loss for the texture, achieving high-fidelity texture reconstruction on the DTU dataset with about 58 FPS on consumer hardware. This disentangled representation enables practical appearance editing while preserving the efficiency and reconstruction quality of 3D-GS, representing a meaningful step toward editable, real-time 3D scene representations.

Abstract

3D Gaussian splatting, emerging as a groundbreaking approach, has drawn increasing attention for its capabilities of high-fidelity reconstruction and real-time rendering. However, it couples the appearance and geometry of the scene within the Gaussian attributes, which hinders the flexibility of editing operations, such as texture swapping. To address this issue, we propose a novel approach, namely Texture-GS, to disentangle the appearance from the geometry by representing it as a 2D texture mapped onto the 3D surface, thereby facilitating appearance editing. Technically, the disentanglement is achieved by our proposed texture mapping module, which consists of a UV mapping MLP to learn the UV coordinates for the 3D Gaussian centers, a local Taylor expansion of the MLP to efficiently approximate the UV coordinates for the ray-Gaussian intersections, and a learnable texture to capture the fine-grained appearance. Extensive experiments on the DTU dataset demonstrate that our method not only facilitates high-fidelity appearance editing but also achieves real-time rendering on consumer-level devices, e.g. a single RTX 2080 Ti GPU.
Paper Structure (18 sections, 19 equations, 14 figures, 2 tables)

This paper contains 18 sections, 19 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Method
  5. Overview
  6. Learning UV Mapping MLP
  7. Learning Texture Value
  8. Ray-Gaussian intersection.
  9. Efficient UV mapping.
  10. Color function.
  11. Experiments
  12. Novel View Synthesis
  13. Texture Editing
  14. Ablation Study
  15. Limitations
  16. ...and 3 more sections

Figures (14)

  • Figure 1: Texture swapping with our method. We propose to disentangle the appearance from the geometry for 3D-GS, thereby facilitating real-time appearance editing such as texture swapping. The rendering speed is shown in each result.
  • Figure 2: Comparison with the straightforward solution. The straightforward solution fails to generate a reasonable and continuous texture, while our method can reconstruct a high-quality texture by considering the intersection of 3D Gaussians and each ray.
  • Figure 3: Visual comparison with previous state-of-the-art editing methods
  • Figure 4: Visual comparison of our method with different numbers of 3D Gaussians
  • Figure 5: Visualization of texture swapping results of our method
  • ...and 9 more figures