Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rendering Anywhere You See: Renderability Field-guided Gaussian Splatting

Xiaofeng Jin, Yan Fang, Matteo Frosi, Jianfei Ge, Jiangjian Xiao, Matteo Matteucci

TL;DR

This work addresses the instability of scene view synthesis under non-uniform, wide-baseline observations by introducing RF-GS, a framework that uses a renderability field to guide pseudo-view sampling and improve coverage. It combines point-projection based inputs with an image restoration network (NAFNet) to convert sparse geometric information into color-consistent views, followed by a two-stage Gaussian primitive optimization that balances global consistency and fine-detail realism. The approach is validated on synthetic and real-world indoor/outdoor data, demonstrating enhanced rendering stability and generalization over prior 3D Gaussian Splatting and related methods, with a new stability metric SDP to capture performance variance. Overall, RF-GS provides a practical pathway toward robust, free-scene rendering suitable for VR/AR and robotics applications.

Abstract

Scene view synthesis, which generates novel views from limited perspectives, is increasingly vital for applications like virtual reality, augmented reality, and robotics. Unlike object-based tasks, such as generating 360° views of a car, scene view synthesis handles entire environments where non-uniform observations pose unique challenges for stable rendering quality. To address this issue, we propose a novel approach: renderability field-guided gaussian splatting (RF-GS). This method quantifies input inhomogeneity through a renderability field, guiding pseudo-view sampling to enhanced visual consistency. To ensure the quality of wide-baseline pseudo-views, we train an image restoration model to map point projections to visible-light styles. Additionally, our validated hybrid data optimization strategy effectively fuses information of pseudo-view angles and source view textures. Comparative experiments on simulated and real-world data show that our method outperforms existing approaches in rendering stability.

Rendering Anywhere You See: Renderability Field-guided Gaussian Splatting

TL;DR

This work addresses the instability of scene view synthesis under non-uniform, wide-baseline observations by introducing RF-GS, a framework that uses a renderability field to guide pseudo-view sampling and improve coverage. It combines point-projection based inputs with an image restoration network (NAFNet) to convert sparse geometric information into color-consistent views, followed by a two-stage Gaussian primitive optimization that balances global consistency and fine-detail realism. The approach is validated on synthetic and real-world indoor/outdoor data, demonstrating enhanced rendering stability and generalization over prior 3D Gaussian Splatting and related methods, with a new stability metric SDP to capture performance variance. Overall, RF-GS provides a practical pathway toward robust, free-scene rendering suitable for VR/AR and robotics applications.

Abstract

Scene view synthesis, which generates novel views from limited perspectives, is increasingly vital for applications like virtual reality, augmented reality, and robotics. Unlike object-based tasks, such as generating 360° views of a car, scene view synthesis handles entire environments where non-uniform observations pose unique challenges for stable rendering quality. To address this issue, we propose a novel approach: renderability field-guided gaussian splatting (RF-GS). This method quantifies input inhomogeneity through a renderability field, guiding pseudo-view sampling to enhanced visual consistency. To ensure the quality of wide-baseline pseudo-views, we train an image restoration model to map point projections to visible-light styles. Additionally, our validated hybrid data optimization strategy effectively fuses information of pseudo-view angles and source view textures. Comparative experiments on simulated and real-world data show that our method outperforms existing approaches in rendering stability.
Paper Structure (15 sections, 7 equations, 20 figures, 3 tables)

This paper contains 15 sections, 7 equations, 20 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Renderability Field Construction
  5. Point Projection Image Restoration
  6. Staged Gaussian Primitive Optimization
  7. Experiments
  8. Methods of Comparison
  9. Dataset Preparation
  10. Metrics of Evaluation
  11. Details of Experiments
  12. Results
  13. Synthetic Data
  14. Real-world Data
  15. Conclusion

Figures (20)

  • Figure 1:
  • Figure 2:
  • Figure 4: Overall pipeline of RF-GS. Given a series of source views and the scene point cloud, our method first calculates three metrics to obtain the renderability value of each viewpoint, which is then resampled to obtain the pseudo-view. The image restoration model is trained based on the source views, and the point cloud projection of the pseudo-view is restored to visible light image during inference. Finally, 3D Gaussian primitives are refined with the mixed data, significantly alleviating rendering artifacts compared to using only ground truth.
  • Figure 5: Observation Conflict Issue. The point cloud captures only the scene’s internal structure, while pseudo-views observe external surfaces to establish co-visibility with source views. However, if no source view captures this exterior region, the co-visible area is falsely inferred.
  • Figure 6: Point cloud image restoration. (a) is the ground truth, (b) is the point cloud image from the viewpoint aligned with the ground truth, (c) the restored image corresponding to the point cloud image in the ground truth view, and (d) the restored image from a new wide baseline viewpoint. The four images on the right display zoomed-in regions, highlighted within the orange boxes in images (a) through (d).
  • ...and 15 more figures