K-Buffers: A Plug-in Method for Enhancing Neural Fields with Multiple Buffers

TL;DR

The paper tackles the trade-off between rendering quality, speed, and model size in neural field representations, focusing on rasterization-based point methods that suffer from noise and holes. It introduces K-Buffers, a plug-in pipeline that renders $K$ z-buffers to produce $K$ pixel-wise feature maps, fuses them with a lightweight KFN in latent space, and decodes to the final image, coupled with a pruning strategy and a tiny feature-rectification MLP. The approach is demonstrated to improve rendering quality for both neural point fields and 3D Gaussian Splatting across multiple benchmarks (e.g., NeRF-Synthetic, ScanNet, DTU, Mip-NeRF 360), while maintaining temporal consistency and keeping memory usage manageable. Overall, K-Buffers provide a scalable, noise-robust enhancement to rasterization-based neural rendering that reduces artifacts and improves detail without large increases in model size.

Abstract

Neural fields are now the central focus of research in 3D vision and computer graphics. Existing methods mainly focus on various scene representations, such as neural points and 3D Gaussians. However, few works have studied the rendering process to enhance the neural fields. In this work, we propose a plug-in method named K-Buffers that leverages multiple buffers to improve the rendering performance. Our method first renders K buffers from scene representations and constructs K pixel-wise feature maps. Then, We introduce a K-Feature Fusion Network (KFN) to merge the K pixel-wise feature maps. Finally, we adopt a feature decoder to generate the rendering image. We also introduce an acceleration strategy to improve rendering speed and quality. We apply our method to well-known radiance field baselines, including neural point fields and 3D Gaussian Splatting (3DGS). Extensive experiments demonstrate that our method effectively enhances the rendering performance of neural point fields and 3DGS.

Figures (9)

• Figure 1: As shown in the figure, our method can simultaneously enhance neural point fields(NPF) and 3DGS.
• Figure 2: The motivation of our method. (a) Previous neural point fields first render the pixel-wise z-buffers and then use a decoder to generate the image from the pixel-wise feature map. However, they are sensitive to the noisy point cloud. (b) A naive solution is to render K z-buffers and use volume rendering to integrate the K colors. However, this solution achieves overfitted results. (c) Our method uses KFN to integrate the K pixel-wise feature maps, significantly enhancing the performance of neural point fields.
• Figure 3: The Overview pipeline with our model. (a) depicts how to obtain the neural descriptors from noisy point clouds. (b) describe how to fuse K pixel-wise features and ultimately obtain the rendering result. (c) illustrates the reason why a single 3D point will generate multiple queried points and how to reduce the number of them by our pruning strategy.
• Figure 4: Illustration of the z-buffer defect. The color of the point cloud has not been used, which is only for visualization purposes.
• Figure 5: We compare the rendered novel views between ours and previous methods. The scenes are, from the top down: kitchen, stump, room, bonsai from Mip-NeRF 360 dataset.