Polarization Wavefront Lidar: Learning Large Scene Reconstruction from Polarized Wavefronts

TL;DR

This work introduces PolLidar, a long-range polarization wavefront lidar that modulates polarization during emission and reception to capture time-resolved polarimetric wavefronts. It combines a Mueller-matrix-based forward model with a CARLA-based simulator and a neural reconstruction network to estimate surface normals, distance, and material properties directly from raw wavefronts. The approach is trained and evaluated on synthetic and real automotive-scale data, achieving 41% lower mean absolute distance error and 53% lower mean angular error for normals compared with baselines. The results demonstrate the value of active polarization sensing for outdoor scene reconstruction and material characterization, with potential impact on autonomous driving and large-scale 3D vision, and code/datasets are open-sourced.

Abstract

Lidar has become a cornerstone sensing modality for 3D vision, especially for large outdoor scenarios and autonomous driving. Conventional lidar sensors are capable of providing centimeter-accurate distance information by emitting laser pulses into a scene and measuring the time-of-flight (ToF) of the reflection. However, the polarization of the received light that depends on the surface orientation and material properties is usually not considered. As such, the polarization modality has the potential to improve scene reconstruction beyond distance measurements. In this work, we introduce a novel long-range polarization wavefront lidar sensor (PolLidar) that modulates the polarization of the emitted and received light. Departing from conventional lidar sensors, PolLidar allows access to the raw time-resolved polarimetric wavefronts. We leverage polarimetric wavefronts to estimate normals, distance, and material properties in outdoor scenarios with a novel learned reconstruction method. To train and evaluate the method, we introduce a simulated and real-world long-range dataset with paired raw lidar data, ground truth distance, and normal maps. We find that the proposed method improves normal and distance reconstruction by 53\% mean angular error and 41\% mean absolute error compared to existing shape-from-polarization (SfP) and ToF methods. Code and data are open-sourced at https://light.princeton.edu/pollidar.

Figures (6)

• Figure 1: PolLidar sensing and scene reconstruction. We design our PolLidar sensor with a unique capability: it modulates the polarization of light during both the emission and reception stages. To this end, a HWP and QWP are used to emit light of a certain polarization, whereas a QWP and LP are used to determine the polarization of the received light. To capture the received signal, we employ an ADC at the APD for precise raw wavefront measurement. This is unlike traditional Lidar systems that primarily focus on distance measurements and do not provide both the polarization characteristics and the wavefront of the light. Subsequently, a novel lidar geometry reconstruction approach predicting normals, distance and material properties is introduced in Sec. \ref{['Sec:Reconstruction']}.
• Figure 2: PolLidar dataset. We capture a long-range polarimetric lidar dataset in typical automotive scenes with object distances up to 100m. On the left is a camera reference image, followed by PolLidar intensity for the horizontally polarized state $\theta_1^{\{1,2,3,4\}}=0$ and sensor-derived ToF distances. On the right, ground truth data from accumulated scans from a Velodyne VLS-128 lidar, providing ToF and surface normals for comparison.
• Figure 3: PolLidar forward model and simulator. Temporal polarimetric reflectance of the scene can be modeled as the sum of specular $\mathbf{M}_s$ and diffuse $\mathbf{M}_d$ reflection. Receiver and emitter of the PolLidar can be described with the Mueller matrices $\mathbf{P}_i$ and $\mathbf{A}_i$ that are functions of the rotation angles $\theta_i^{\{1,2,3,4\}}$ of HWP, QWPs and LP, respectively. We employ the resulting PolLidar forward model in a simulator based on CARLA that generates synthetic polarimetric raw wavefronts. To this end, we extract material properties and normals from CARLA and feed them into the forward model. The resulting temporal wavefronts are subsequently downsampled in spatial dimension to model beam divergence and noise is added to simulate APD and ADC.
• Figure 4: Neural polarization wavefront lidar reconstruction. We capture raw polarization wavefronts of the scene $\mathbf{I}$. We apply a peak-based segmentation technique to obtain a sliced polarization wavefront $\Tilde{\mathbf{I}}$ and distance priors $\mathbf{d}$. Via ellipsometric reconstruction, we estimate a sliced Mueller matrix $\mathbf{H}_{\mathrm{meas}}$. Finally, we concatenate all the polarization priors with viewing direction $\mathbf{V}$ as the input to a neural network predicting distance and normals for the scene. We supervised the network with a normal loss $\mathcal{L}_{\mathrm{normal}}$ and a distance loss $\mathcal{L}_{\mathrm{dist}}$.
• Figure 5: Qualitative evaluation on synthetic data. Baek et al. Baek:PolarToF:2022 is unable to reconstruct normals in areas with low DoP, e.g., walls of buildings facing the sensor. PCA Zhou2018open3d applied in this setting are strongly dependent on point cloud density. Thus, distant poles and cars in the second row cannot be reconstructed accurately. The proposed approach leverages polarization cues to reconstruct normals in sparse regions and is robust against low DoP areas. We estimate accurate material properties for different surfaces and objects (right).