On the Error Analysis of 3D Gaussian Splatting and an Optimal Projection Strategy

TL;DR

This work analyzes projection errors in 3D Gaussian Splatting that arise from the local affine projection of Gaussian primitives. It derives an error measure $\epsilon(\boldsymbol{\mu}')$ via the Taylor remainder $R_1(\mathbf{x}')$ and shows how the error depends on the Gaussian mean, enabling identification of extrema. Building on this, it introduces Optimal Gaussian Splatting, projecting each Gaussian onto a plane tangent to the unit sphere along the line to the camera center, yielding an optimal projection function $\varphi_p$ with Jacobian $\mathbf{J_p}$ that can accommodate various camera models. A Unit Sphere Based Rasterizer is then proposed to render by evaluating tangent-plane Gaussians and alpha-blending them, preserving real-time performance. Experiments across 13 real scenes demonstrate reduced artifacts and improved rendering quality compared to the original 3D-GS and competitive results with NeRF-based methods, especially at short focal lengths, highlighting practical robustness and broad applicability.

Abstract

3D Gaussian Splatting has garnered extensive attention and application in real-time neural rendering. Concurrently, concerns have been raised about the limitations of this technology in aspects such as point cloud storage, performance, and robustness in sparse viewpoints, leading to various improvements. However, there has been a notable lack of attention to the fundamental problem of projection errors introduced by the local affine approximation inherent in the splatting itself, and the consequential impact of these errors on the quality of photo-realistic rendering. This paper addresses the projection error function of 3D Gaussian Splatting, commencing with the residual error from the first-order Taylor expansion of the projection function. The analysis establishes a correlation between the error and the Gaussian mean position. Subsequently, leveraging function optimization theory, this paper analyzes the function's minima to provide an optimal projection strategy for Gaussian Splatting referred to Optimal Gaussian Splatting, which can accommodate a variety of camera models. Experimental validation further confirms that this projection methodology reduces artifacts, resulting in a more convincingly realistic rendering.

Figures (12)

• Figure 1: By minimizing the projection error through error analysis, we have achieved an improvement in the rendering image quality compared to the original 3D-GS kerbl20233d. These images are cropped from the complete images of the Train scene, in order to be highlighted.
• Figure 2: Visualization of the 3D Gaussian Splatting error function $\epsilon\left(\theta_{\mu},\phi_{\mu}\right)$ under two distinct domains. The domain of the function is $\left\{\left(\theta_{\mu},\phi_{\mu}\right)\mid \theta_{\mu}\in\left[-\lambda\pi/4, \lambda\pi/4\right] ~\land~\phi_{\mu}\in\left[-\lambda\pi/4, \lambda\pi/4\right] \right\}$ where $\left(\theta_{\mu},\phi_{\mu}\right)$ represents the polar coordinate of the 3D Gaussian mean and $\lambda$ represents the scaling factor of the domain.
• Figure 3: Illustration of the rendering pipeline for our Optimal Gaussian Splatting and the projection of 3D-GS kerbl20233d. The blue box depicts the projection process of the original 3D-GS, which straightforwardly projects all Gaussians onto the same projection plane. In contrast, the red box illustrates our approach, where we project individual Gaussians onto corresponding tangent planes.
• Figure 4: We show comparisons of our method to previous methods and the corresponding ground truth images from held-out test views. The scenes are, from the top down: Truck from Tanks$\&$Temples knapitsch2017tanks; Playroom from the Deep Blending dataset hedman2018deep and Bonsai, Counter from Mip-NeRF360 dataset barron2022mip. Differences in quality highlighted by arrows/insets.
• Figure 5: We derive the projection error function with respect to image coordinates $(u, v)$ and focal length through the transformation between image and polar coordinates $(\theta_{\mu}, \phi_{\mu})$. We visualize this function, showing 3D-GS rendered images with long and short focal lengths, followed by the corresponding error functions.