Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering

TL;DR

<3-5 sentence high-level summary>

Abstract

3D Gaussians have recently emerged as a highly efficient representation for 3D reconstruction and rendering. Despite its high rendering quality and speed at high resolutions, they both deteriorate drastically when rendered at lower resolutions or from far away camera position. During low resolution or far away rendering, the pixel size of the image can fall below the Nyquist frequency compared to the screen size of each splatted 3D Gaussian and leads to aliasing effect. The rendering is also drastically slowed down by the sequential alpha blending of more splatted Gaussians per pixel. To address these issues, we propose a multi-scale 3D Gaussian splatting algorithm, which maintains Gaussians at different scales to represent the same scene. Higher-resolution images are rendered with more small Gaussians, and lower-resolution images are rendered with fewer larger Gaussians. With similar training time, our algorithm can achieve 13\%-66\% PSNR and 160\%-2400\% rendering speed improvement at 4$\times$-128$\times$ scale rendering on Mip-NeRF360 dataset compared to the single scale 3D Gaussian splitting. Our code and more results are available on our project website https://jokeryan.github.io/projects/ms-gs/

Figures (14)

• Figure 1: Overall pipeline of our algorithm. At the early stage of training (left), small Gaussians below certain size threshold in each voxel are aggregated, enlarged and inserted into the scene at different resolution scale. During rendering (right), the multi-scale Gaussians of the appropriate "pixel coverage" at the current render resolution are selected for rendering. If the rendering resolution scale equals to the scale of the Gaussians, the expected "pixel coverage" range of the Gaussians are updated accordingly.
• Figure 2: Pixel coverage of a 3D Gaussian is its horizontal or vertical size, whichever is smaller measured by the level set.
• Figure 3: Missing parts caused by naive small Gaussian filtering at different resolution scales.
• Figure 4: Large Gaussians are created by aggregating the small Gaussians in each voxel below the pixel coverage threshold, and then enlarged by the pixel coverage multiplier.
• Figure 5: Based on the rendering resolution, the current pixel coverage of a Gaussian relative to its minimum and maximum pixel coverages determines whether it is selected for rendering.