Projecting Gaussian Ellipsoids While Avoiding Affine Projection Approximation

TL;DR

Experiments over multiple widely adopted benchmark datasets show that the proposed ellipsoid-based projection method can enhance the rendering quality of 3D Gaussian Splatting and its extensions.

Abstract

Recently, 3D Gaussian Splatting has dominated novel-view synthesis with its real-time rendering speed and state-of-the-art rendering quality. However, during the rendering process, the use of the Jacobian of the affine approximation of the projection transformation leads to inevitable errors, resulting in blurriness, artifacts and a lack of scene consistency in the final rendered images. To address this issue, we introduce an ellipsoid-based projection method to calculate the projection of Gaussian ellipsoid onto the image plane, which is the primitive of 3D Gaussian Splatting. As our proposed ellipsoid-based projection method cannot handle Gaussian ellipsoids with camera origins inside them or parts lying below $z=0$ plane in the camera space, we designed a pre-filtering strategy. Experiments over multiple widely adopted benchmark datasets show that our ellipsoid-based projection method can enhance the rendering quality of 3D Gaussian Splatting and its extensions.

Figures (4)

• Figure 1: Our method achieves a comprehensive improvement in rendering quality and rendering speed compared to 3D Gaussian splatting (3DGS) 3DGS. We propose an ellipsoid-based projection method to replace the Jacobian of the affine approximation of the projection transformation in 3DGS. Our ellipsoid-based projection method can be applied to any 3DGS-based work to enhance rendering quality. This figure shows the rendering results of applying our method to 3DGS and Mip-Splatting anti-aliasing1 in the scene bicycle of Mip-NeRF360 dataset render_quality2, in which the rendering quality is enhanced with less blur and artifacts.
• Figure 2: Ellipsoid-based Projection Method. We first derive the ellipsoid equation based on the covariance matrix of the 3D Gaussian function. Then, using the method in \ref{['sec:3.2']}, we obtain the equation of a cone with its vertex at the camera origin and tangent to the ellipsoid. Finally, we calculate the intersection line between the cone and the image plane, which gives the equation of the projection of the ellipsoid.
• Figure 3: Pre-filtering Strategy. There are two types of Gaussian ellipsoids that need to be filtered out in advance. Otherwise, the system may fail to converge. The first type is Gaussian ellipsoids that contain the camera origin within them, represented by the blue ellipsoid in the figure. The second type consists of Gaussian ellipsoids with portions located below the $z=0$ plane in the camera space. The projection of these ellipsoids results in parabolas or hyperbolas, as shown by the orange ellipsoid in the figure.
• Figure 4: We demonstrated the rendering results of applying our ellipsoid-based projection method to 3DGS and Mip-Splatting, resulting in less blur and artifact and better scene consistency.