Adapting Depth Anything to Adverse Imaging Conditions with Events

TL;DR

This work addresses depth estimation under adverse imaging conditions by bridging the strengths of depth foundation models with event-based sensing. It introduces ADAE, a cross-modal adapter that injects event features into a frozen Depth Anything model, augmented by Entropy-Aware Spatial Fusion for illumination-robust fusion and Motion-Guided Temporal Correction for blur mitigation, formalized through a joint optimization with L_ADAE = $\lambda_1 \mathcal{L}_{gt} + \lambda_2 \mathcal{L}_{s} + \lambda_3 \mathcal{L}_{t}$. Key contributions include adaptive fusion via patch-level entropy, event-guided foreground-background disentanglement using dense event-based optical flow, and a practical training scheme with a pretraining stage for the event encoder. Extensive synthetic and real-world experiments demonstrate improved depth accuracy and boundary fidelity while preserving the generalization of the foundation model, including zero-shot performance on MVSEC, EVRB, and NCER. The results indicate that combining data-driven generalization with event-based signal enhancement yields robust depth estimation in challenging environments, with code to be released after acceptance.

Abstract

Robust depth estimation under dynamic and adverse lighting conditions is essential for robotic systems. Currently, depth foundation models, such as Depth Anything, achieve great success in ideal scenes but remain challenging under adverse imaging conditions such as extreme illumination and motion blur. These degradations corrupt the visual signals of frame cameras, weakening the discriminative features of frame-based depths across the spatial and temporal dimensions. Typically, existing approaches incorporate event cameras to leverage their high dynamic range and temporal resolution, aiming to compensate for corrupted frame features. However, such specialized fusion models are predominantly trained from scratch on domain-specific datasets, thereby failing to inherit the open-world knowledge and robust generalization inherent to foundation models. In this work, we propose ADAE, an event-guided spatiotemporal fusion framework for Depth Anything in degraded scenes. Our design is guided by two key insights: 1) Entropy-Aware Spatial Fusion. We adaptively merge frame-based and event-based features using an information entropy strategy to indicate illumination-induced degradation. 2) Motion-Guided Temporal Correction. We resort to the event-based motion cue to recalibrate ambiguous features in blurred regions. Under our unified framework, the two components are complementary to each other and jointly enhance Depth Anything under adverse imaging conditions. Extensive experiments have been performed to verify the superiority of the proposed method. Our code will be released upon acceptance.

Figures (6)

• Figure 1: Comparison of four depth estimation paradigms. (a) Frame-based specialized models are trained on domain-specific datasets and suffer from poor generalization. (b) Frame-based foundation models leverage large-scale datasets for rich knowledge and strong generalization, but fail in adverse imaging conditions. (c) Event-frame fusion models enhance robustness but are also specialized and lack generalization. (d) Our event-enhanced foundation model synergizes the strong generalization of a frozen foundation model (b) with the signal-level robustness of event-based fusion (c).
• Figure 2: Zero-shot depth prediction on MVSEC zhu2018multivehicle, EVRB kim2024cmta, and NCER cho2023non datasets. We compare our method (ADAE) with a representative event-frame fusion method (RAMNet) and the foundation model (Depth Anything V2). The top two rows show scenes with extreme illumination (over- and underexposure), while the bottom two rows contain motion blur. Our method produces more structurally complete and detailed depth maps.
• Figure 3: Overview of our proposed ADAE framework. ADAE introduces a cross-modal adapter that integrates event features into the frozen Depth Anything model, and further enhances it through Entropy-Aware Spatial Fusion (EASF) and Motion-Guided Temporal Correction (MGTC). The EASF leverages information entropy as a proxy for signal quality to adaptively fuse frame and event features, addressing degradation from extreme illumination. The MGTC utilizes optical flow estimated from events to guide the disentanglement of foreground and background features corrupted by motion blur.
• Figure 4: Motivation of Entropy-Aware Spatial Fusion (EASF). Under extreme illumination, frame suffer from over- or underexposure, while events remain robust but sparse. We observe that information entropy reflects signal reliability in both modalities. This motivates us to adjust fusion weights based on patch-wise entropy comparisons between frame and event modalities, enabling spatially adaptive feature integration under adverse lighting conditions.
• Figure 5: Motivation of Motion-Guided Temporal Correction (MGTC). We estimate depth on blurred and sharp frames using Depth Anything and visualize their feature distributions via t-SNE. In blurred regions, foreground and background features are entangled, while sharp regions exhibit clear separation. This motivates us to leverage temporally dense event-based optical flow, which captures past and future boundary motions, to localize foreground and background regions within motion-blurred areas. This localization then guides the disentanglement of corrupted features, restoring distinct structural boundaries.