Geometry-Informed Distance Candidate Selection for Adaptive Lightweight Omnidirectional Stereo Vision with Fisheye Images

TL;DR

This work addresses the high computational cost of omnidirectional distance estimation using fisheye cameras by introducing geometry-informed distance candidate selection that enables accurate results with a small, deployment-adaptive candidate set. The GI approach computes distance candidates to yield near-constant feature displacement between successive candidates, facilitating effective interpolation even as camera extrinsics or the number of cameras change. Empirical results show that GI candidates improve performance when layouts vary and enable real-time operation with hardware acceleration on embedded platforms, while maintaining competitive accuracy with far fewer candidates. The authors release model variants and a large synthetic dataset to facilitate adoption and further research in deployment-friendly omnidirectional stereo vision.

Abstract

Multi-view stereo omnidirectional distance estimation usually needs to build a cost volume with many hypothetical distance candidates. The cost volume building process is often computationally heavy considering the limited resources a mobile robot has. We propose a new geometry-informed way of distance candidates selection method which enables the use of a very small number of candidates and reduces the computational cost. We demonstrate the use of the geometry-informed candidates in a set of model variants. We find that by adjusting the candidates during robot deployment, our geometry-informed distance candidates also improve a pre-trained model's accuracy if the extrinsics or the number of cameras changes. Without any re-training or fine-tuning, our models outperform models trained with evenly distributed distance candidates. Models are also released as hardware-accelerated versions with a new dedicated large-scale dataset. The project page, code, and dataset can be found at https://theairlab.org/gicandidates/ .

Figures (10)

• Figure 1: After training, using our geometry-informed (GI) distance candidate distribution, the baseline distance between cameras can be changed and the model's performance can be restored without fine-tuning.
• Figure 2: Model Overview. The model takes three fisheye images as input during training and performs learned feature extraction with a shared feature extractor, builds a cost volume with spherical sweeping, and regularizes the distance with a 3D U-Net won2019omnimvs.
• Figure 3: Camera Configuration for the evaluation board. Three fisheye cameras are mounted pointing upwards in a triangular formation. A LiDAR, unused for this study, introduces self-occlusions.
• Figure 4: GI and EV candidates for different camera spacings. EV candidates approximate constant feature displacement steps for small spacings (baselines), but result in highly uneven steps in large spacings. GI candidates generate constant displacement steps as a function of camera spacing.
• Figure 5: Sphere-sweeping geometry. We pick distance candidates that result in constant steps in ray angle corresponding to constant displacements in the projected feature.