Robust and Real-time Surface Normal Estimation from Stereo Disparities using Affine Transformations

TL;DR

This work tackles fast, accurate surface normal estimation from rectified stereo disparities by exploiting affine correspondences. A GPU-friendly, geometry-based pipeline computes normals from affine parameters, with affine parameters estimated from disparity via a convolution-accelerated least-squares scheme and robust border handling through adaptive region estimation. The method is deterministic, learning-free, and yields dense oriented point clouds; experiments show real-time performance and improved accuracy over PCA, particularly under noise and at moderate kernel sizes. The approach promises practical impact for real-time 3D reconstruction and robotic perception, with shader code to support reproducibility.

Abstract

This work introduces a novel method for surface normal estimation from rectified stereo image pairs, leveraging affine transformations derived from disparity values to achieve fast and accurate results. We demonstrate how the rectification of stereo image pairs simplifies the process of surface normal estimation by reducing computational complexity. To address noise reduction, we develop a custom algorithm inspired by convolutional operations, tailored to process disparity data efficiently. We also introduce adaptive heuristic techniques for efficiently detecting connected surface components within the images, further improving the robustness of the method. By integrating these methods, we construct a surface normal estimator that is both fast and accurate, producing a dense, oriented point cloud as the final output. Our method is validated using both simulated environments and real-world stereo images from the Middlebury and Cityscapes datasets, demonstrating significant improvements in real-time performance and accuracy when implemented on a GPU. Upon acceptance, the shader source code will be made publicly available to facilitate further research and reproducibility.

Figures (11)

• Figure 1: Estimated surface normals are drawn in a vehicle-mounted camera image from the Cityscape dataset Cordts2016Cityscapes. For the road, normals are vertical, reconstruction of those can significantly improve e.g. the segmentation of road from other objects. Normals are estimated by the proposed approach.
• Figure 2: Geometric setup for rectified stereo. It is represented by baseline $b$ and common focal length $f$. The inputs are the disparity values $d_i$ coming from the projection of spatial points $\mathbf X_i$.
• Figure 3: Two corresponding regions (squares) of the stereo image pair. The region in the left image is estimated by applying an affine transformation to its corresponding pair in the right image. This figure shows the original and the estimated regions overlapped. ($\mathbf c,\mathbf c'$) and ($\mathbf p,\mathbf p'$) denote two correspondences between the two regions with respective disparities $d_c$ and $d_p$. $\mathbf c,\mathbf c'$ are central pixels of the two regions, the pixels at where the affine transformation is estimated and normal calculated. $\mathbf v=\mathbf p-\mathbf c$, $\mathbf v'=\mathbf p'-\mathbf c'$. Note that $\mathbf v'=\mathbf{A} \mathbf v$.
• Figure 4: An example of a $3\times 3$ kernel for computing $a_1$. The listed coefficients $s_i$ and their corresponding vectors $\mathbf{v_i}^T$ are listed in the same cell.
• Figure 5: Visualization of the selected (green) pixels by performing star-fill from C in eight directions.