URS-NeRF: Unordered Rolling Shutter Bundle Adjustment for Neural Radiance Fields

TL;DR

A novel rolling shutter bundle adjustment method for neural radiance fields (NeRF), which utilizes the unordered rolling shutter images to obtain the implicit 3D representation and recovers the physical formation of RS images by estimating camera poses and velocities, thereby removing the input constraints on sequential data.

Abstract

We propose a novel rolling shutter bundle adjustment method for neural radiance fields (NeRF), which utilizes the unordered rolling shutter (RS) images to obtain the implicit 3D representation. Existing NeRF methods suffer from low-quality images and inaccurate initial camera poses due to the RS effect in the image, whereas, the previous method that incorporates the RS into NeRF requires strict sequential data input, limiting its widespread applicability. In constant, our method recovers the physical formation of RS images by estimating camera poses and velocities, thereby removing the input constraints on sequential data. Moreover, we adopt a coarse-to-fine training strategy, in which the RS epipolar constraints of the pairwise frames in the scene graph are used to detect the camera poses that fall into local minima. The poses detected as outliers are corrected by the interpolation method with neighboring poses. The experimental results validate the effectiveness of our method over state-of-the-art works and demonstrate that the reconstruction of 3D representations is not constrained by the requirement of video sequence input.

Figures (10)

• Figure 1: The illustration of different NeRF BA settings. (a) BARF lin2021barf works on unordered global shutter (GS) images, but is unsuitable for images with rolling shutter distortions. (b) USB-NeRF li2023usb requires the input to be strictly sequential video frames, which lacks the generality in BA. (c) Our method works on unordered images with rolling shutter distortion.
• Figure 2: Overall pipeline of our proposed framework. We adopt a coarse-to-fine strategy to train rolling shutter images, and the scene graph is used to detect and correct the estimated poses that belong to outliers. Refer to the text for more details.
• Figure 3: (a) Image formation models of a RS camera (top) and a GS camera (bottom). (b) Final image shapes of different motion modes for RS camera. It demonstrates that each row of a rolling shutter image is captured at different timestamps, and would thus lead to different image distortions if the image is captured by a moving camera.
• Figure 4: Visual comparison of the initial and optimized camera poses for Tri-MipRF, Tri-MipRF-BA, URS-NeRF-wo, and URS-NeRF on Traj1-medium-large setting. URS-NeRF successfully realigns all the camera frames, while some estimated poses of Tri-MipRF-BA and URS-NeRF-wo get stuck at suboptimal solutions. The original Trimip-RF lacks the bundle adjustment, resulting in the poorest performance.
• Figure 5: Qualitative comparisons with the unordered images on the WHU-RS datasets. The second row consists of disparity maps between rendered images and ground truth, where darker areas indicate better performance. The experiments demonstrate a significant improvement in both rendering quality and reduction of rolling shutter effects with our method.