Online Refractive Camera Model Calibration in Visual Inertial Odometry

TL;DR

This work tackles the challenge of visual-inertial odometry through refractive media by introducing a general refractive camera model that incorporates Snell's law, refractive and lens distortions, and an online estimation of the medium's refractive index $n$. By embedding $n$ as an augmentable state in an IEKF-based VIO (ROVIO) framework, the method jointly estimates odometry and the refractive index from air-calibrated cameras, using a photometric error with a carefully derived Jacobian chain for iterative optimization. The authors derive forward and inverse refractive mappings, their Jacobians, and a sensitivity heuristic to improve robustness in degenerate or low-texture settings, validating convergence of $n$ across liquids and initializations in underwater pool experiments. The approach achieves on-par VIO performance in refractive media without medium-specific calibration, and a public dataset is released to enable broader evaluation and adoption.

Abstract

This paper presents a general refractive camera model and online co-estimation of odometry and the refractive index of unknown media. This enables operation in diverse and varying refractive fluids, given only the camera calibration in air. The refractive index is estimated online as a state variable of a monocular visual-inertial odometry framework in an iterative formulation using the proposed camera model. The method was verified on data collected using an underwater robot traversing inside a pool. The evaluations demonstrate convergence to the ideal refractive index for water despite significant perturbations in the initialization. Simultaneously, the approach enables on-par visual-inertial odometry performance in refractive media without prior knowledge of the refractive index or requirement of medium-specific camera calibration.

Figures (6)

• Figure 1: Instance of the conducted experimental studies employing a Remotely Operated Vehicle integrating a time-synchronized camera/IMU system navigating in a pool subject to diverse light conditions. The proposed approach enables online estimation of the medium's refractive index and thus adjusts the camera model employed within visual-inertial odometry.
• Figure 2: Visualization of a ray from a point $\mathbf{p}_{\mathcal{C}}$ in the refractive media (e.g. water), undergoing refractive distortion modelled by $\mathbf{g}_{r}$, followed by lens distortion modelled by $\mathbf{g}_{l}$. Lastly, the point is projected onto camera array, modelled by $\mathbf{K}$, at pixel coordinate $\mathbf{u}$. The thickness of the refractive interface $\delta_{w}$ and the distance of the interface from the camera lens $\delta_{lw}$ are assumed to be small.
• Figure 3: Detailed evaluation on a rectangular trajectory with good ambient light conditions (Trajectory 1). Top Left: The first lap of the estimated trajectory, given refractive index $n$ initialization of $1.35$, with path colorized based on the refractive index absolute error from the ideal value of the refractive index of water $n=1.33$. The bottom left plot shows the full trajectory for $370~\textrm{sec}$ along with the ground truth (dashed blue). Top right: the plot showing refractive index $n$ vs. time with initialization of $n$ varying from $1.31$ to $1.35$. Bottom right: The comparison of odometry against ground truth vs. time for online estimation with initialization from $n$ equal to $1.35$, fixed value of $n$ at $1.33$, and calibration of the camera directly inside the water of the same pool using an equidistant model is water. The map from the accumulated point cloud is generated using lipson2021raftstereo for visualization.
• Figure 4: The top-down plots of trajectory on left show the colorized path based on the absolute difference from the ideal value of the refractive index for water $n=1.33$ when initialized with $1.35$ (purple line in the right plot starting from $1.35$). The path is colored according to the error bar shown at the bottom. The blue circle is the starting point of the robot. For clarity of visualization, only the first lap of motion is shown. The refractive index vs. time plots on the right show, for the complete mission, the convergence of refractive index $n$ from perturbations ranging from $1.31$ to $1.35$. The lack of visual texture results in the sparseness of the point cloud generated using lipson2021raftstereo for low-light conditions.
• Figure 5: Collective plots for trajectories comparing against ground truth for an initial $n$ of $1.35$. On the left, odometry estimates for trajectories 1-3 (good visibility). Similarly, the plots on the right show the estimates in the low-light trajectories 4-6.