LED: Light Enhanced Depth Estimation at Night

TL;DR

Nighttime depth estimation remains difficult due to low illumination and distribution shifts. LED introduces a pattern-projection cue from vehicle HD headlights to guide monocular depth models, improving accuracy across multiple architectures with data-efficient learning. The approach is validated on a synthetic nighttime dataset (NSDD) and a real-world dataset, showing gains inside and beyond the illuminated regions and better cross-domain robustness. By releasing NSDD and demonstrating compatibility with existing architectures, LED offers a practical, low-cost augmentation to nightly perception systems for autonomous driving.

Abstract

Nighttime camera-based depth estimation is a highly challenging task, especially for autonomous driving applications, where accurate depth perception is essential for ensuring safe navigation. Models trained on daytime data often fail in the absence of precise but costly LiDAR. Even vision foundation models trained on large amounts of data are unreliable in low-light conditions. In this work, we aim to improve the reliability of perception systems at night time. To this end, we introduce Light Enhanced Depth (LED), a novel, cost-effective approach that significantly improves depth estimation in low-light environments by harnessing a pattern projected by high definition headlights available in modern vehicles. LED leads to significant performance boosts across multiple depth-estimation architectures (encoder-decoder, Adabins, DepthFormer, Depth Anything V2) both on synthetic and real datasets. Furthermore, increased performances beyond illuminated areas reveal a holistic enhancement in scene understanding. Finally, we release the Nighttime Synthetic Drive Dataset, a synthetic and photo-realistic nighttime dataset, which comprises 49,990 comprehensively annotated images.

Figures (13)

• Figure 1: Light Enhanced Depth (LED) is a novel method that harnesses high-definition headlights' pattern projected onto the scene to enhance nighttime depth estimation from RGB images. We release a synthetic nighttime dataset with high beam and pattern-illuminated images, along with comprehensive ground truth annotations, to advance research in nighttime perception.
• Figure 2: Pattern deformation in the scene: Left: For explanatory purpose, we project the pattern on a depth image. It demonstrates trapezoidal deformations on horizontal surfaces and undistorted squares on vertical surfaces. Complex deformations on the car reveal insights into its geometry. Right: Pattern projected on a wall at 10 m (darker) and 100 m (lighter). Square sizes increases with distance.
• Figure 3: Nighttime Synthetic Drive Dataset examples: (A) depicts HD pattern and (B) high-beam illumination. Ground truth annotations include dense depth maps, semantic segmentation, instance segmentation labels and bounding boxes.
• Figure 4: Simulation of HD pattern: Top-left: control matrix of the HD headlight. Bottom-left: photometry considering aberrations created by the headlight lens. Right: Resulting image using the photometry. The area outlined in red is the region of interest.
• Figure 5: Qualitative results on NSDD. LED results exhibit higher precision, and more accurate object boundaries and shapes compared to HB. Red boxes indicate enhanced regions.