Relative Pose for Nonrigid Multi-Perspective Cameras: The Static Case

TL;DR

This work addresses relative pose estimation for static non-rigid multi-perspective cameras (NR-MPCs) by modeling gravity-induced deformation with a cantilever beam. It integrates the deformation into the generalized epipolar framework via gravity-dependent extrinsics $\mathbf{w}_i({^{\mathcal{B}}}\mathbf{g})$ and $\mathbf{W}_i({^{\mathcal{B}}}\mathbf{g})$, and uses Plücker line coordinates with a generalized essential matrix $\mathcal{G}$ to constrain correspondences. Two solver strategies are proposed: a vision-only approach and a gravity-informed approach that leverages an IMU prior, both aiming to recover $\mathbf{t}$, $\mathbf{R}$, and the gravity direction $^{\mathcal{B}}\mathbf{g}$ (8 DOF total) through robust estimation and nonlinear bundle adjustment. Synthetic results demonstrate gravity observability and quantify the impact of bar stiffness, noise, and outliers, while real-data experiments confirm practical viability on indoor NR-MPC systems. The findings suggest NR-MPCs can serve as passive inertial sensors, exposing latent gravity states through purely geometric constraints and without additional sensors, with implications for exteroceptive sensing in robotics and AR/VR.

Abstract

Multi-perspective cameras with potentially non-overlapping fields of view have become an important exteroceptive sensing modality in a number of applications such as intelligent vehicles, drones, and mixed reality headsets. In this work, we challenge one of the basic assumptions made in these scenarios, which is that the multi-camera rig is rigid. More specifically, we are considering the problem of estimating the relative pose between a static non-rigid rig in different spatial orientations while taking into account the effect of gravity onto the system. The deformable physical connections between each camera and the body center are approximated by a simple cantilever model, and inserted into the generalized epipolar constraint. Our results lead us to the important insight that the latent parameters of the deformation model, meaning the gravity vector in both views, become observable. We present a concise analysis of the observability of all variables based on noise, outliers, and rig rigidity for two different algorithms. The first one is a vision-only alternative, while the second one makes use of additional gravity measurements. To conclude, we demonstrate the ability to sense gravity in a real-world example, and discuss practical implications.

Figures (12)

• Figure 1: Illustrations of Nonrigid multi-perspective cameras.
• Figure 2: Geometry of the traditional, rigid MPC problem. Points are observed in two different body frame poses $\mathcal{B}$ and $\mathcal{B}'$ by cameras with extrinsics $(\mathbf{v}_j,\mathbf{V}_j)$. Example camera frames are indicated in green, and direction vectors $\mathbf{f}$ are all expressed inside these frames.
• Figure 3: Case of an NR-MPC where the bars connecting to the body center are deforming under the influence of gravity.
• Figure 4: Location and orientation of the intermediate frame $\mathcal{C}_i$. The canonical position of the camera in $\mathcal{C}_i$ is located along the $z$ axis.
• Figure 5: Deformations $\delta_{ix}$, $\delta_{iy}$, $\theta_{ix}$ and $\theta_{iy}$ caused by the first and the second component of ${^{\mathcal{C}_i}}\mathbf{g}$.