Through the Curved Cover: Synthesizing Cover Aberrated Scenes with Refractive Field

TL;DR

This work introduces SynthCover to enable novel view synthesis through protective covers for downstream extended reality applications and demonstrates the method's ability to accurately model scenes viewed through protective covers, achieving a significant improvement in rendering quality compared to prior methods.

Abstract

Recent extended reality headsets and field robots have adopted covers to protect the front-facing cameras from environmental hazards and falls. The surface irregularities on the cover can lead to optical aberrations like blurring and non-parametric distortions. Novel view synthesis methods like NeRF and 3D Gaussian Splatting are ill-equipped to synthesize from sequences with optical aberrations. To address this challenge, we introduce SynthCover to enable novel view synthesis through protective covers for downstream extended reality applications. SynthCover employs a Refractive Field that estimates the cover's geometry, enabling precise analytical calculation of refracted rays. Experiments on synthetic and real-world scenes demonstrate our method's ability to accurately model scenes viewed through protective covers, achieving a significant improvement in rendering quality compared to prior methods. We also show that the model can adjust well to various cover geometries with synthetic sequences captured with covers of different surface curvatures. To motivate further studies on this problem, we provide the benchmarked dataset containing real and synthetic walkable scenes captured with protective cover optical aberrations.

Figures (6)

• Figure 1: This figure illustrates the four modules of the proposed framework: (a) Refractive Field: Estimates surface normals and ray-to-cover distances, using Snell's Law to calculate ray offsets. (b) Ray Sampling: Follows refracted paths, calculating positional offsets for rays. (c) Radiance Field Rendering: Renders the scene using the sampled points.
• Figure 2: (a) Refraction from curved cover distorts light paths and causes rendering artifacts. (b) Our method addresses this by modeling the cover geometry and analytically bending the rays conforming to Snell's physical refraction law.
• Figure 3: Sample images from the CurvedCover dataset and the capturing rig.
• Figure 4: (a) Surface reconstruction of the curved cover. (b) Simulation setup for recreating the distortion induced by the curved cover.
• Figure 5: This figure compares NeRF variants, including ours. Ground truth images are shown in the first column. All methods use RealityCepture for initial calibration. Our approach shows consistent accuracy across datasets.