LiFMCR: Dataset and Benchmark for Light Field Multi-Camera Registration

TL;DR

LiFMCR introduces a public benchmark for multi-camera plenoptic registration by pairing synchronized MLA-based light-field sequences from two Raytrix R32 cameras with high-precision $6$-DoF ground-truth poses from a Vicon system. The authors present a complete calibration-and-registration pipeline and implement two baseline methods: a $3D$ RANSAC-based transformation estimation and a plenoptic PnP approach that uses a single light-field image, both accounting for the plenoptic camera model. Experimental results show consistent relative pose accuracy (rotation around a few degrees, translation around tens of millimeters) and a measurable but systematic offset in absolute pose due to frame-reference differences between the Vicon ground truth and camera optical centers. This dataset and its baselines enable rigorous evaluation of multi-view light-field registration and support advances in perception tasks such as SLAM, NVS, and 3D scene understanding with plenoptic cameras.

Abstract

We present LiFMCR, a novel dataset for the registration of multiple micro lens array (MLA)-based light field cameras. While existing light field datasets are limited to single-camera setups and typically lack external ground truth, LiFMCR provides synchronized image sequences from two high-resolution Raytrix R32 plenoptic cameras, together with high-precision 6-degrees of freedom (DoF) poses recorded by a Vicon motion capture system. This unique combination enables rigorous evaluation of multi-camera light field registration methods. As a baseline, we provide two complementary registration approaches: a robust 3D transformation estimation via a RANSAC-based method using cross-view point clouds, and a plenoptic PnP algorithm estimating extrinsic 6-DoF poses from single light field images. Both explicitly integrate the plenoptic camera model, enabling accurate and scalable multi-camera registration. Experiments show strong alignment with the ground truth, supporting reliable multi-view light field processing. Project page: https://lifmcr.github.io/

Figures (10)

• Figure 1: Pipeline for registering camera images and estimating 6-DOF extrinsics between views. Camera 0 serves as reference and other cameras are registered using one of the two proposed methods. Note that the point clouds from cameras 1 to X are required only for the RANSAC method, not for the plenoptic PnP algorithm.
• Figure 1: Graphical representation of camera trajectories and frame poses for the two Raytrix R32 cameras in the main sequences of each scene.
• Figure 2: Pipeline of the two proposed camera registration algorithms: the method in Fig. \ref{['fig:pipeline_RANSAC']} uses 3D pose estimation via RANSAC, and the method in Fig. \ref{['fig:pipeline_PnP']} uses PnP.
• Figure 2: Graphical representation of the parameters in the mla calibration file.
• Figure 3: Central perspective projection of the virtual image onto a common image plane. The points are projected along lines through the main lens center (blue) onto a plane at a distance of $2B$ from the MLA, instead of using horizontal projection (orange). Here, $B$ is the distance between the MLA and the sensor, $b_{L0}$ is the distance between the main lens and the MLA, and $v_3$ is the virtual depth of point $X_{V3}'$.