Content-Aware Texturing for Gaussian Splatting

TL;DR

This work addresses the inefficiency of traditional Gaussian Splatting in capturing fine appearance by introducing content-aware textures that are attached to 2D Gaussian primitives. Textures are defined with fixed world-space texel sizes and adapt during optimization through downscale/upscale operations, enabling finer detail where needed while reducing texture memory via resolution-aware primitive management. The method decouples appearance from geometry, supports progressive texel-size adaptation, and uses regularisation to maintain compact parameterization, achieving competitive image quality with fewer parameters than prior textured-Gaussian approaches. Extensive experiments across multiple datasets demonstrate robust performance, offering a versatile approach to texture-aware 3D reconstruction and real-time rendering.

Abstract

Gaussian Splatting has become the method of choice for 3D reconstruction and real-time rendering of captured real scenes. However, fine appearance details need to be represented as a large number of small Gaussian primitives, which can be wasteful when geometry and appearance exhibit different frequency characteristics. Inspired by the long tradition of texture mapping, we propose to use texture to represent detailed appearance where possible. Our main focus is to incorporate per-primitive texture maps that adapt to the scene in a principled manner during Gaussian Splatting optimization. We do this by proposing a new appearance representation for 2D Gaussian primitives with textures where the size of a texel is bounded by the image sampling frequency and adapted to the content of the input images. We achieve this by adaptively upscaling or downscaling the texture resolution during optimization. In addition, our approach enables control of the number of primitives during optimization based on texture resolution. We show that our approach performs favorably in image quality and total number of parameters used compared to alternative solutions for textured Gaussian primitives. Project page: https://repo-sam.inria.fr/nerphys/gs-texturing/

Figures (6)

• Figure 1: Difference between two mappings for a primitive having learned a particular texture. Top row: a naive approach distorts the appearance after the primitive undergoes scaling. Bottom row: in our approach, as texel size is fixed in world space, scaling the primitive only reveals more part of the underlying texture, preserving existing content.
• Figure 2: We illustrate the downscale and upscale process used.
• Figure 3: Left to right:The primitive has been upscaled to a resolution greater than $\tau_{tr}$ in both axes. Our splitting approach creates four new primitives, each with half the scale and texture resolution in both axes.
• Figure 4: The primitive has high error, and successive upscalings result in a texture resolution above threshold. In this case, the error is geometric rather than due to appearance. Our algorithm performs a split, and further optimization will match the geometry.
• Figure 5: We provide renderings from one scene per dataset for every method, trained with default settings. Our method is able to reconstruct high frequency details on images, even while using fewer parameters compared to the other methods.