RPBG: Towards Robust Neural Point-based Graphics in the Wild

TL;DR

RPBG tackles the fragility of neural point-based rendering in real-world, in-the-wild scenes by identifying core failure modes in NPBG. It introduces a downgrade-aware neural renderer (DAC), a lightweight environment model, and a point-triangulation augmentation strategy, enabling end-to-end collaborative optimization that generalizes across unbounded, inside-out, large-scale, and sparse-view data. The approach achieves superior perceptual quality (PSNR/SSIM/LPIPS) and robustness compared to RF-based and prior point-based methods, with significant efficiency advantages in large-scale settings. These contributions provide a practical, scalable, and robust alternative for neural point-based graphics in real-world applications, with code available for replication.

Abstract

Point-based representations have recently gained popularity in novel view synthesis, for their unique advantages, e.g., intuitive geometric representation, simple manipulation, and faster convergence. However, based on our observation, these point-based neural re-rendering methods are only expected to perform well under ideal conditions and suffer from noisy, patchy points and unbounded scenes, which are challenging to handle but defacto common in real applications. To this end, we revisit one such influential method, known as Neural Point-based Graphics (NPBG), as our baseline, and propose Robust Point-based Graphics (RPBG). We in-depth analyze the factors that prevent NPBG from achieving satisfactory renderings on generic datasets, and accordingly reform the pipeline to make it more robust to varying datasets in-the-wild. Inspired by the practices in image restoration, we greatly enhance the neural renderer to enable the attention-based correction of point visibility and the inpainting of incomplete rasterization, with only acceptable overheads. We also seek for a simple and lightweight alternative for environment modeling and an iterative method to alleviate the problem of poor geometry. By thorough evaluation on a wide range of datasets with different shooting conditions and camera trajectories, RPBG stably outperforms the baseline by a large margin, and exhibits its great robustness over state-of-the-art NeRF-based variants. Code available at https://github.com/QT-Zhu/RPBG.

Figures (14)

• Figure 1: Left: RPBG manages to achieve all-round good re-renderings (PSNR plotted) across generic datasets over the baseline aliev2020neural as well as state-of-the-art RF-based methods turki2022megawang2023f2. Right: We demonstrate the point clouds (with camera trajectories visualized) and corresponding re-rendered novel views of the representative scenes, revealing the great robustness and scalability of RPBG. Zoom in for best view.
• Figure 2: Typically challenging scenes in T&T dataset knapitsch2017tanks for NPBG. Top:Auditorium, where the walls and ceilings are extremely sparse. Bottom:Museum, where the point sparsity makes the occlusion and visibility complicated.
• Figure 3: The overall pipeline of RPBG. Point Triangulation: We first triangulate a 3D proxy for re-rendering with posed images, with its geometry-bounded neural texture initialized. Point Rasterization: The points are raterized to the given camera in a non-differentiable manner. By indexing the texture with the fragment, we obtain the neural buffer. A learnable point-size neural texture $\mathbf{T}_{\textrm{env}}$ is also optimized. Neural Rendering: The restoration from downgraded neural buffer to photo-realistic images is performed by a CNN. The network and the neural texture are optimized end-to-end by image-level losses. An offline point cloud augmentation strategy is introduced to alleviate the problem of patchy triangulation under challenging conditions.
• Figure 4: The architecture of the downgrade-aware neural renderer in RPBG, with some conventional modules omitted. From the visualized attention map, DAC manages to adaptively handle the severely erroneous point visibility.
• Figure 5: Visualized comparisons over varying scenes. From top to bottom:sky and hydrant of the Free dataset wang2023f2, Courtroom and Train of T&T dataset knapitsch2017tanks, and DayaTemple and MemorialHall of GigaMVS dataset zhang2021gigamvs. We include the results of RPBG (Ours), Gaussian Splatting kerbl20233d, NPBG aliev2020neural, and F$^2$-NeRF wang2023f2 for comparison. Zoom in for best view.