Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields

TL;DR

GeFu tackles generalizable neural radiance field view synthesis by addressing geometry reconstruction and rendering with three core modules: Adaptive Cost Aggregation (ACA) to stabilize geometry under occlusions, Spatial-View Aggregator (SVA) to encode 3D context in descriptors, and Consistency-Aware Fusion (CAF) to unify the strengths of two rendering strategies. The method operates in a coarse-to-fine pipeline where coarse geometry guides fine sampling and ACA is informed by coarse-stage features, enabling robust geometry reasoning and high-fidelity rendering without per-scene fine-tuning. Empirical results on DTU, Real Forward-facing llff, and NeRF Synthetic show state-of-the-art generalization and competitive depth maps, with CAF contributing substantial gains and the combination of ACA, SVA, and CAF delivering strong performance across challenging regions. The work offers a practical, modular approach to improving generalizable NeRFs, achieving notable improvements in occlusion and reflective regions while maintaining efficiency gains in the render pipeline.

Abstract

Generalizable NeRF aims to synthesize novel views for unseen scenes. Common practices involve constructing variance-based cost volumes for geometry reconstruction and encoding 3D descriptors for decoding novel views. However, existing methods show limited generalization ability in challenging conditions due to inaccurate geometry, sub-optimal descriptors, and decoding strategies. We address these issues point by point. First, we find the variance-based cost volume exhibits failure patterns as the features of pixels corresponding to the same point can be inconsistent across different views due to occlusions or reflections. We introduce an Adaptive Cost Aggregation (ACA) approach to amplify the contribution of consistent pixel pairs and suppress inconsistent ones. Unlike previous methods that solely fuse 2D features into descriptors, our approach introduces a Spatial-View Aggregator (SVA) to incorporate 3D context into descriptors through spatial and inter-view interaction. When decoding the descriptors, we observe the two existing decoding strategies excel in different areas, which are complementary. A Consistency-Aware Fusion (CAF) strategy is proposed to leverage the advantages of both. We incorporate the above ACA, SVA, and CAF into a coarse-to-fine framework, termed Geometry-aware Reconstruction and Fusion-refined Rendering (GeFu). GeFu attains state-of-the-art performance across multiple datasets. Code is available at https://github.com/TQTQliu/GeFu .

Figures (19)

• Figure 1: Comparison with existing methods. (a) With three input source views, our generalizable model synthesizes novel views with higher quality than existing methods mvsnerfenerf in the severe occluded area. (b) Circle area represents inference time. The X-axis represents the PSNR on the DTU dataset dtu and the Y-axis represents the PSNR on the Real Forward-facing dataset llff. Our method attains state-of-the-art performance.
• Figure 2: Comparison of two rendering strategies. (a) The view obtained using the blending approach that combines color values from source views. (b) The view obtained using the regression approach that directly regresses color values from features. (c) Accuracy comparison between two rendering strategies. The former strategy performs better in the white regions, while worse in the green ones.
• Figure 3: The overview of GeFu. In the reconstruction phase, we first infer the geometry from the constructed cost volume, and the geometry guides us to further re-sample 3D points around the surface. For each sampled point, the warped features from source images are aggregated and then fed into our proposed Spatial-View Aggregator (SVA) to learn spatial and inter-view context-aware descriptors $f_p$. In the rendering phase, we apply two decoding strategies to obtain two intermediate views and fuse them into the final target view in an adaptive way, termed Consistency-Aware Fusion (CAF). Our pipeline adopts a coarse-to-fine architecture, the geometry from the coarse stage ($l_s=1$) guides the sampling at the fine stage ($l_s > 1$), and the features from the coarse stage are transferred to the fine stage for ACA to improve geometry estimation. Our network is trained end-to-end using only RGB images.
• Figure 4: Qualitative comparison of rendering quality with state-of-the-art methods mvsnerfenerfmatchnerf under generalization and three input views settings.
• Figure 5: Qualitative comparison of depth maps with enerf.