LiFCal: Online Light Field Camera Calibration via Bundle Adjustment

TL;DR

LiFCal tackles online, target-free calibration of MLA-based plenoptic cameras by introducing a plenoptic camera model integrated into a full online bundle adjustment pipeline. The method initializes with a pinhole-based estimate and then refines intrinsics, extrinsics, and 3D scene points directly from micro-image coordinates, using a Levenberg–Marquardt optimization. It achieves accuracy comparable to state-of-the-art target-based calibration, enabling reliable metric depth estimation and integration with RGB-D SLAM without scene-scale priors. This yields a practical, scalable solution for deploying plenoptic cameras in robotics, AR/VR, and autonomous systems where targets are unavailable or impractical.

Abstract

We propose LiFCal, a novel geometric online calibration pipeline for MLA-based light field cameras. LiFCal accurately determines model parameters from a moving camera sequence without precise calibration targets, integrating arbitrary metric scaling constraints. It optimizes intrinsic parameters of the light field camera model, the 3D coordinates of a sparse set of scene points and camera poses in a single bundle adjustment defined directly on micro image points. We show that LiFCal can reliably and repeatably calibrate a focused plenoptic camera using different input sequences, providing intrinsic camera parameters extremely close to state-of-the-art methods, while offering two main advantages: it can be applied in a target-free scene, and it is implemented online in a complete and continuous pipeline. Furthermore, we demonstrate the quality of the obtained camera parameters in downstream tasks like depth estimation and SLAM. Webpage: https://lifcal.github.io/

Figures (17)

• Figure 1: Process overview: Raw images undergo camera calibration via plenoptic bundle adjustment. This yields a metric camera model used to compute a totally focused image and depth map from new raw images, enabling accurate metric depth measurement.
• Figure 1:
• Figure 2: Focused plenoptic camera in Galilean mode. Parameter $f_L$ is the main lens focal length, $b_{L0}$ the distance between main lens and mla, $B$ the distance between mla and sensor, $b$ the distance between mla and virtual image, $b_L$ the distance between main lens and virtual image and $z_C$ the distance between real object and main lens. The virtual image created by the main lens is a mirror image of the real scene.
• Figure 3: Projection modeling inside the plenoptic camera. (\ref{['fig:projectionDepth']}) Projection of a virtual image point $X_V'=[x_V', y_V', z_V' = v]^T$ to a raw image point $X_R=[x_R, y_R]^T$ through the micro lens center $C_{ML}=[c_{MLx}, c_{MLy}]^T$. The virtual image point is given in dimensionless coordinates, i.e. image distance $b=v \cdot B$ is normalized by the distance $B$. (\ref{['fig:centersProj']}) Projection from the micro lens centers $C_{ML}$ to the micro image centers $C_I$.
• Figure 4: Flowchart of the online calibration algorithm for the focused plenoptic camera. Images are acquired from the camera and are first used to initialize the parameters. Next, a complete bundle adjustment for the plenoptic camera model is performed.