HDGS: Textured 2D Gaussian Splatting for Enhanced Scene Rendering

TL;DR

HDGS tackles the limitations of 2D Gaussian Splatting in rendering from arbitrary viewpoints by introducing per-surfel texture maps, per-ray sorting, and Fisher pruning to disentangle appearance from geometry and maintain efficiency. It also introduces frustum-based sampling to mitigate aliasing in high-frequency appearances across resolutions. The method demonstrates state-of-the-art rendering quality on standard benchmarks and a texture-rich dataset, showing improved detail preservation and anti-aliasing while maintaining geometric fidelity. Limitations include higher computational cost due to per-ray sorting and texture indexing, suggesting room for optimization for real-time deployment.

Abstract

Recent advancements in neural rendering, particularly 2D Gaussian Splatting (2DGS), have shown promising results for jointly reconstructing fine appearance and geometry by leveraging 2D Gaussian surfels. However, current methods face significant challenges when rendering at arbitrary viewpoints, such as anti-aliasing for down-sampled rendering, and texture detail preservation for high-resolution rendering. We proposed a novel method to align the 2D surfels with texture maps and augment it with per-ray depth sorting and fisher-based pruning for rendering consistency and efficiency. With correct order, per-surfel texture maps significantly improve the capabilities to capture fine details. Additionally, to render high-fidelity details in varying viewpoints, we designed a frustum-based sampling method to mitigate the aliasing artifacts. Experimental results on benchmarks and our custom texture-rich dataset demonstrate that our method surpasses existing techniques, particularly in detail preservation and anti-aliasing.

Figures (10)

• Figure 1: Overview of the improvements in our method over 2D Gaussian Splatting (2DGS). First two columns: Our frustum-based sampling technique effectively mitigates aliasing for high-frequency appearance rendering at reduced resolutions (1/4 and 1/8). Third column: Our approach yields enhanced geometric consistency with smoother surface normals. Last column: Our method disentangles geometry from appearance by utilizing a per-Gaussian texture map, allowing for rich detail preservation that 2DGS cannot resolve.
• Figure 2: Our proposed method for high-quality scene reconstruction leverages the explicit per-ray depth and 2D ray-surfel intersection coordinates of 2D Gaussian Splatting. The left image illustrates our coarse-to-fine pipeline, which begins with 2DGS training, proceeds through Fisher pruning, and ends with per-surfel texture mapping for enhanced visual fidelity. The right image highlights our design of per-ray sorting strategy with primitive texture: though the center of surfel $p_2$ is closer to the screen than $p_1$, our method sorts the Gaussians correctly as $z_1$ and $z_2$ in the depth order.
• Figure 3: Frustum Sampling. The average density of the sampled primitive decreases as the surfel moves farther from the camera or as the sampling resolution decreases, effectively acting as a low-pass filter for anti-aliasing.
• Figure 4: Comparison of our method and 2DGS at reduced resolutions (1/2, 1/4, 1/8) on NeRF synthetic dataset mildenhall2020nerf and Mip-NeRF 360 dataset barron2022mip. Our approach effectively mitigates dilation artifacts and aliasing problems in 2DGS. Note that the 1/8 rendering of the microphone by 2DGS has the same image scale as above, whose dilation aliasing is considerably severe.
• Figure 5: Quanlitative comparison of novel view rendering on five custom scenes featuring materials with intricate details of our method with 3D Gaussian Splatting (3DGS) and 2D Gaussian Splatting (2DGS). Each row showcases a distinct texture-rich material novel view rendering: 1) the fabric texture on a bag, 2) the paper surface of a cardboard box, 3) the text and texture of the book cover, 4) the label details on a box, 5) fine texture of a face mask. The four columns display results from 3DGS, 2DGS, our method, and the ground truth. Across all scenes, our approach consistently captures finer details, demonstrating superior texture fidelity compared to 3DGS and 2DGS.