Dense Scene Reconstruction from Light-Field Images Affected by Rolling Shutter

TL;DR

The paper tackles dense depth estimation from light-field images acquired with rolling shutter by introducing a two-stage approach that first establishes a RS-agnostic dense 3D representation via 2D Gaussian Splatting, and then jointly estimates 3D motion to compensate for RS-induced deformation. It reframes RS distortion through a differentiable render-and-compare pipeline that links 3D geometry, RS motion, and appearance, yielding RS-compensated disparity and central-view images. A new RS-LF dataset, RSLF+, is designed to enable thorough evaluation on textured scenes with occlusion masks, and the authors release code and models to promote reproducibility. The method achieves dense, RS-aware reconstructions without priors and demonstrates clear advantages over baselines in both RS and GS contexts, offering practical impact for robust LF-based depth estimation in real-world rolling shutter cameras.

Abstract

This paper presents a dense depth estimation approach from light-field (LF) images that is able to compensate for strong rolling shutter (RS) effects. Our method estimates RS compensated views and dense RS compensated disparity maps. We present a two-stage method based on a 2D Gaussians Splatting that allows for a ``render and compare" strategy with a point cloud formulation. In the first stage, a subset of sub-aperture images is used to estimate an RS agnostic 3D shape that is related to the scene target shape ``up to a motion". In the second stage, the deformation of the 3D shape is computed by estimating an admissible camera motion. We demonstrate the effectiveness and advantages of this approach through several experiments conducted for different scenes and types of motions. Due to lack of suitable datasets for evaluation, we also present a new carefully designed synthetic dataset of RS LF images. The source code, trained models and dataset will be made publicly available at: https://github.com/ICB-Vision-AI/DenseRSLF

Figures (14)

• Figure 1: Dense depth and motion compensation example from a light-field image affected by RS distortions. The RS deformed central sub-aperture view (used as input) is shown in the top left. The top right view shows the corresponding view with no RS effect. Our method results are shown in the second row, with the rendered compensated appearance model (bottom left) and disparity (bottom right) generated from the set of learned 2D Gaussians.
• Figure 1: A simplified plenoptic camera scheme. The main difference to a standard camera is the addition of a microlens array between the main lens and the sensor.
• Figure 2: Overview of the main components of our two-stage method. In the first stage, we estimate a 2D Gaussian Splatting representation from a sub-set of SAIs. In the second stage, using the ability to render and compare thanks to the 2D Gaussian Splatting representation, we estimate the motion parameters in order to recover an undistorted representation of the scene.
• Figure 2: The unprocessed image obtained with a plenoptic camera. We can see in the detail the multitude of micro-images produced by the microlenses. Images from [3].
• Figure 3: Properties of a rolling shutter light-field image (RSLF) with motion. An RSLF sensor provides 4D data possessing the joint properties of an RS sensor (top left view) and of a Light-Field sensor (top right). It outputs RS affected sub-aperture images (SAI).