NFL-BA: Near-Field Light Bundle Adjustment for SLAM in Dynamic Lighting

TL;DR

This work tackles SLAM performance degradation under dynamic near-field lighting when the light source is co-located with the camera, a common situation in endoscopy and subterranean robotics. It introduces Near-Field Lighting Bundle Adjustment (NFL-BA), a plug-in loss that explicitly models near-field illumination within neural rendering-based SLAM by decomposing surface appearance into albedo and a shading term $PPS(\cdot)$ driven by relative camera-surface geometry. The NFL-BA loss replaces the standard Photometric BA in existing SLAM systems, improving camera tracking and 3D mapping across endoscopy datasets (e.g., C3VD) and indoor scenes, with substantial gains such as $\sim$37% reduction in translation error and improved mapping quality. The approach is demonstrated as a versatile, drop-in module compatible with multiple SLAM backbones (e.g., MonoGS, EndoGSLAM, NICE-SLAM), highlighting the practical impact for autonomous navigation and visualization in challenging lighting conditions. Limitations include handling of specular and sub-surface effects, which are proposed as directions for future work.

Abstract

Simultaneous Localization and Mapping (SLAM) systems typically assume static, distant illumination; however, many real-world scenarios, such as endoscopy, subterranean robotics, and search & rescue in collapsed environments, require agents to operate with a co-located light and camera in the absence of external lighting. In such cases, dynamic near-field lighting introduces strong, view-dependent shading that significantly degrades SLAM performance. We introduce Near-Field Lighting Bundle Adjustment Loss (NFL-BA) which explicitly models near-field lighting as a part of Bundle Adjustment loss and enables better performance for scenes captured with dynamic lighting. NFL-BA can be integrated into neural rendering-based SLAM systems with implicit or explicit scene representations. Our evaluations mainly focus on endoscopy procedure where SLAM can enable autonomous navigation, guidance to unsurveyed regions, blindspot detections, and 3D visualizations, which can significantly improve patient outcomes and endoscopy experience for both physicians and patients. Replacing Photometric Bundle Adjustment loss of SLAM systems with NFL-BA leads to significant improvement in camera tracking, 37% for MonoGS and 14% for EndoGS, and leads to state-of-the-art camera tracking and mapping performance on the C3VD colonoscopy dataset. Further evaluation on indoor scenes captured with phone camera with flashlight turned on, also demonstrate significant improvement in SLAM performance due to NFL-BA. See results at https://asdunnbe.github.io/NFL-BA/

Figures (10)

• Figure 1: NFL-BA enhances tracking and mapping in neural rendering-based SLAM (e.g., MonoGS MonoGS) by explicitly modeling dynamic near-field lighting, with applications in endoscopy.
• Figure 2: MonoGS performance under (1) distant static lighting and (2) dynamic near-field lighting from a co-located flashlight. Standard photometric BA performs well under static lighting but fails under dynamic lighting, degrading both trajectory and map quality. NFL-BA restores performance under dynamic lighting, matching the quality of the static-light setup.
• Figure 3: Illustration of our key idea. As the co-located light and camera, moves through the scene, different 3D Gaussians on the surface receive different intensities of light (red arrow), dependent on the relative distance and orientation between the 3D Gaussian and the camera.
• Figure 4: Image Formation Validation. We show that C3VD images captured with a real endoscope conform to our co-located light-camera and zero attenuation $\beta$ image formation model, as indicated by very low per-pixel scale-invariant MSE between the original image and the reconstructed image with masked-out specular regions.
• Figure 5: Camera tracking improvement over EndoGSLAM Wan_EndoGSLAM_MICCAI2024. Replacing the Photo-BA loss (in blue) with NFL-BA loss (in red) significantly improves camera tracking for different depth initialization. Average tracking error ATE$_t$ for each sequence is reported in the inset. (zoom for details)