Removing Multiple Hybrid Adverse Weather in Video via a Unified Model

TL;DR

This work tackles the problem of removing multiple hybrid adverse weather effects from video using a single, adaptable model. It introduces UniWRV, featuring a Weather Prior Guided Module (WPGM) for spatial adaptivity and a Dynamic Routing Aggregation (DRA) for temporal fusion, along with the HWVideo synthetic dataset to benchmark all-in-one restoration under 15 hybrid conditions. Key contributions include the priors-as-prompts approach with dedicated loss terms, a sparse, efficient routing mechanism, and extensive experiments showing strong performance on HWVideo, real-world data, and other generic video degradations. The proposed framework offers robust cross-condition generalization and practical pathways for deploying all-in-one video restoration in real-world applications such as autonomous systems and surveillance.

Abstract

Videos captured under real-world adverse weather conditions typically suffer from uncertain hybrid weather artifacts with heterogeneous degradation distributions. However, existing algorithms only excel at specific single degradation distributions due to limited adaption capacity and have to deal with different weather degradations with separately trained models, thus may fail to handle real-world stochastic weather scenarios. Besides, the model training is also infeasible due to the lack of paired video data to characterize the coexistence of multiple weather. To ameliorate the aforementioned issue, we propose a novel unified model, dubbed UniWRV, to remove multiple heterogeneous video weather degradations in an all-in-one fashion. Specifically, to tackle degenerate spatial feature heterogeneity, we propose a tailored weather prior guided module that queries exclusive priors for different instances as prompts to steer spatial feature characterization. To tackle degenerate temporal feature heterogeneity, we propose a dynamic routing aggregation module that can automatically select optimal fusion paths for different instances to dynamically integrate temporal features. Additionally, we managed to construct a new synthetic video dataset, termed HWVideo, for learning and benchmarking multiple hybrid adverse weather removal, which contains 15 hybrid weather conditions with a total of 1500 adverse-weather/clean paired video clips. Real-world hybrid weather videos are also collected for evaluating model generalizability. Comprehensive experiments demonstrate that our UniWRV exhibits robust and superior adaptation capability in multiple heterogeneous degradations learning scenarios, including various generic video restoration tasks beyond weather removal.

Figures (15)

• Figure 1: Overview of video adverse weather removal frameworks. (a) Separate networks designed for specific weather; (b) generic networks but with task-specific weights; (c) our proposed UniWRV framework. In contrast to existing approaches that aim to tackle different weather types with different model instances, UniWRV can handle multiple hybrid adverse weather videos with heterogeneous degradation distributions via a unified model, thus enjoying better flexibility and practicality in realistic applications.
• Figure 2: Feature visualization of three heterogeneous weather conditions (i.e., rain, snow, and rain+snow). The conventional method RVRT liang2022recurrent only perform well on single degradation learning however is unable to cope with unseen weather degradations. Directly applying RVRT to learn multiple weather degradations will lead to inadequate and ill-focused feature representation due to limited adaptation capacity. In contrast, our proposed UniWRV exhibits excellent adaptation capacity to heterogeneous degradations, and thus can adaptively and comprehensively handle given weather degradations.
• Figure 3: Architecture of UniWRV for unified multiple hybrid weather removal from video, which takes three adjacent frames with unknown weather artifacts as input and restores the clean mid-frame. Our UniWRV consists of three sub-modules: a feature extraction network and a feature reconstruction network built upon weather prior guided module (WPGM) that perform adaptive spatial features processing, a dynamic routing aggregation module (DRA) that performs adaptive multi-frame temporal features fusion.
• Figure 4: (a): Weather prior guided module that performs weather adaptive feature extraction via prior query. (b): Deformable multi-frame attention block that performs multi-frame feature fusion. (c): Path controller that determines the most appropriate fusion routing of multi-frame features.
• Figure 5: Illustration of the proposed modify weights routing scheme. Instead of routing via redundant multi-node structure, we advocate routing cross lightweight modify weights and performing only one projection calculation with the modified parameter.