Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environments

TL;DR

This work addresses the failure of classic photometric bundle adjustment in non-Lambertian environments by introducing a physically-based PBA that weights photometric errors using illumination, material, and light-path priors. It decomposes radiance into components via $r \approx T_{\text{Light}} \cdot T_{\text{BRDF}}$, with a weight $\delta_{\mathbf{p}i} = \exp(-\theta |r - r'|)$ to down-weight inconsistent pixels, and leverages environment maps predicted from a point cloud alongside a transformer-based multi-view material estimator. It also introduces SpecularRooms, a dataset with ground-truth illumination and material to enable evaluation of non-Lambertian SLAM/PBA methods. Experimental results show that the proposed PB-PBA yields higher accuracy than Lambertian and other robust methods across multiple trajectories, aided by dedicated material and illumination estimation pipelines.

Abstract

Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. However, the assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. The photometric inconsistency significantly affects the reliability of existing PBA methods. To solve this problem, we propose a novel physically-based PBA method. Specifically, we introduce the physically-based weights regarding material, illumination, and light path. These weights distinguish the pixel pairs with different levels of photometric inconsistency. We also design corresponding models for material estimation based on sequential images and illumination estimation based on point clouds. In addition, we establish the first SLAM-related dataset of non-Lambertian scenes with complete ground truth of illumination and material. Extensive experiments demonstrated that our PBA method outperforms existing approaches in accuracy.

Figures (7)

• Figure 1: Effectiveness of our PBA method in a non-Lambertian environment. DSM-PBA dsm improves the accuracy of ORB-SLAM2 mur2017orb, and our PBA method is more accurate than DSM-PBA. The color bar shows the magnitude of the absolute trajectory error.
• Figure 2: Phsically-based reflection model based on point light source. The incident direction $\boldsymbol{\alpha}_k$ is associated with the radiance $I_k$. The radiance received by a 3D point $\mathbf{P}$ is then reflected as the radiance $R_k$ along the reflective direction $\boldsymbol{\beta}$. The pixel $\mathbf{p}$ receives the sum of reflective radiances along the direction $\boldsymbol{\beta}$.
• Figure 3: Material estimation pipeline. Our approach uses the current frame and $K-1$ previous frames to predict the roughness of the current frame. A CNN encoder extracts features from these $K$ frames. A transformer and a decoder process these features to predict the roughness of the current frame.
• Figure 4: Illumination estimation pipeline. The colored point cloud is fed into the point encoder, which extracts per-point features. These features are then projected and used as inputs for the image decoder. The image decoder then produces a coarse output. The coarse output is combined with a color-depth map in the RefineNet, yielding a refined output.
• Figure 5: Trajectories estimated by ORB-SLAM2 mur2017orb and various PBA methods on three image sequences of our SpecularRooms dataset. The colored and black lines denote the estimated and ground truth trajectories, respectively. The color bar indicates the magnitude of the absolute trajectory error.