Textured Gaussians for Enhanced 3D Scene Appearance Modeling

TL;DR

This work extends 3D Gaussian Splatting by attaching per-Gaussian texture maps (alpha, RGB, or RGBA) to each Gaussian, enabling spatially varying color and opacity and substantially increasing expressive power without increasing the primitive count. The approach leverages ray-Gaussian intersection, plane-based UV mapping, and joint optimization of textures with the Gaussian parameters in a two-stage training regime, achieving superior novel-view synthesis across object- and scene-level benchmarks. Notably, alpha-only textures offer strong gains with smaller model budgets, while full RGBA textures deliver the best overall fidelity, albeit with higher memory costs and training time. The method maintains the advantageous properties of 3DGS—explicit 3D primitives and fast rendering—while enabling finer appearance detail and complex geometries, with potential extensions to dynamic scenes and 3D texture representations.

Abstract

3D Gaussian Splatting (3DGS) has recently emerged as a state-of-the-art 3D reconstruction and rendering technique due to its high-quality results and fast training and rendering time. However, pixels covered by the same Gaussian are always shaded in the same color up to a Gaussian falloff scaling factor. Furthermore, the finest geometric detail any individual Gaussian can represent is a simple ellipsoid. These properties of 3DGS greatly limit the expressivity of individual Gaussian primitives. To address these issues, we draw inspiration from texture and alpha mapping in traditional graphics and integrate it with 3DGS. Specifically, we propose a new generalized Gaussian appearance representation that augments each Gaussian with alpha~(A), RGB, or RGBA texture maps to model spatially varying color and opacity across the extent of each Gaussian. As such, each Gaussian can represent a richer set of texture patterns and geometric structures, instead of just a single color and ellipsoid as in naive Gaussian Splatting. Surprisingly, we found that the expressivity of Gaussians can be greatly improved by using alpha-only texture maps, and further augmenting Gaussians with RGB texture maps achieves the highest expressivity. We validate our method on a wide variety of standard benchmark datasets and our own custom captures at both the object and scene levels. We demonstrate image quality improvements over existing methods while using a similar or lower number of Gaussians.

Figures (41)

• Figure 1: Textured Gaussians compared to 3D Gaussian Splatting (3DGS). Our RGBA Textured Gaussians enhance 3D scene appearance modeling, leading to improved rendering quality while using the same number of Gaussians compared to 3DGS kerbl3Dgaussians. Above, we show that Textured Gaussians faithfully reconstruct the fine details of scenes.
• Figure 2: Variants of our Textured Gaussians model. Textured Gaussians encapsulate four kinds of color and opacity spatial variations. The top row of the figure shows the texture map associated with each Gaussian, and the bottom row shows the rendered Textured Gaussians. The constant-color and constant-alpha model (no textures) corresponds to the original 3DGS formation, which can only represent a single color up to a Gaussian falloff factor within the Gaussian extent. Textured Gaussians can already model spatially varying colors using only alpha textures since each pixel can be alpha-composited differently. The model achieves maximum expressivity when leveraging the full RGBA texture map, where each Gaussian is capable of representing complex shapes and high frequency textures.
• Figure 3: Textured Gaussians model pipeline. Our method consists of three major components: ray-Gaussian intersection, RGBA texture mapping, and a generalized Gaussian appearance model. To render the color of a pixel $\mathbf{p}$, we first trace a ray from the camera center $\mathbf{o}$ to the pixel to intersect with 3D Gaussians in the scene. Then, we query texture and alpha values, $\mathbf{c}_\text{tex}$ and $\alpha_\text{tex}$, from the per-Gaussian RGBA texture maps using the ray-Gaussian intersection point $\mathbf{x}$. Finally, given the retrieved spatially varying color and alpha values, we alpha-composite the color and alpha values of Gaussians that are hit by the pixel ray using the generalized Gaussian appearance model.
• Figure 4: NVS performance using varying numbers of Gaussians. When using the same number of Gaussians as 3DGS$^*$, our Textured Gaussians achieve better novel view synthesis results on all five benchmark datasets in PSNR. Here, we show the quantitative performance of 3DGS* and Textured Gaussians models optimized with varying number of Gaussians on the three scene-level datasets.
• Figure 5: Qualitative NVS results of benchmark datasets. Novel-view rendering results for both object-level and scene-level standard benchmark datasets. Given the same small number of Gaussians (on average 2.5k and 39k Gaussians for object-level and scene-level datasets, respectively), 3DGS$^*$ fails to reconstruct high-frequency textures and complex shapes while our RGBA Textured Gaussians model succeeds. Please refer to the Appendix for results of our models optimized using varying number of Gaussians.