Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation

TL;DR

This work explores synthetic degraded domain as an intermediate bridge between clean and real degraded domains, and proposes a cumulative homogeneous-heterogeneous adaptation framework for real adverse weather optical flow that can progressively and explicitly transfer knowledge from clean scenes to real adverse weather.

Abstract

Optical flow has made great progress in clean scenes, while suffers degradation under adverse weather due to the violation of the brightness constancy and gradient continuity assumptions of optical flow. Typically, existing methods mainly adopt domain adaptation to transfer motion knowledge from clean to degraded domain through one-stage adaptation. However, this direct adaptation is ineffective, since there exists a large gap due to adverse weather and scene style between clean and real degraded domains. Moreover, even within the degraded domain itself, static weather (e.g., fog) and dynamic weather (e.g., rain) have different impacts on optical flow. To address above issues, we explore synthetic degraded domain as an intermediate bridge between clean and real degraded domains, and propose a cumulative homogeneous-heterogeneous adaptation framework for real adverse weather optical flow. Specifically, for clean-degraded transfer, our key insight is that static weather possesses the depth-association homogeneous feature which does not change the intrinsic motion of the scene, while dynamic weather additionally introduces the heterogeneous feature which results in a significant boundary discrepancy in warp errors between clean and degraded domains. For synthetic-real transfer, we figure out that cost volume correlation shares a similar statistical histogram between synthetic and real degraded domains, benefiting to holistically aligning the homogeneous correlation distribution for synthetic-real knowledge distillation. Under this unified framework, the proposed method can progressively and explicitly transfer knowledge from clean scenes to real adverse weather. In addition, we further collect a real adverse weather dataset with manually annotated optical flow labels and perform extensive experiments to verify the superiority of the proposed method.

Figures (17)

• Figure 1: Optical flow visualization of the proposed method CH$^2$DA-Flow under adverse weather. The proposed method can well estimate the high-quality optical flows in various degraded conditions, including dynamic weather (e.g., rain and snow) and static weather (e.g., veiling and fog).
• Figure 2: Illustration of adaptation paradigms for adverse weather optical flow. (a) Direct adaptation. (b) Cumulative adaptation. (c) Adverse weather optical flows. (d) Cumulative homogeneous-heterogeneous adaptation. Direct adaptation mainly transfers motion knowledge from clean to degraded domain via one-stage adaptation.However, this direct adaptation usually suffers from domain shift due to the large domain gap between clean and real degraded domains. To address this issue, we propose a cumulative adaptation framework, which introduces an intermediate domain to close the large gap, making the motion features of the intermediate domain aligned with the features of the two domains, respectively. Considering that dynamic weather (e.g., rain) and static weather (e.g., fog) have different impacts on optical flow even within the degraded domain, we further extend the cumulative adaptation framework into the cumulative homogeneous-heterogeneous adaptation framework, which not only aligns the cross-domain homogeneous motion features of the scene, but also strips away the heterogeneous motion features of the dynamic degradation.
• Figure 3: The architecture of the proposed CH$^2$DA-Flow method mainly contains clean-degraded motion adaptation (CDMA) and synthetic-real motion adaptation (SRMA). During the CDMA stage, we design the depth association homogeneous motion adaptation for static weather, and the warp error heterogeneous boundary adaptation for dynamic weather, thus directionally transferring motion knowledge from the clean domain to the synthetic degraded domain. During the SRMA stage, we propose the cost volume homogeneous correlation adaptation, which builds the correlation distribution holistic alignment module to distill motion knowledge of the synthetic degraded domain to the real degraded domain.
• Figure 4: Visualization of optical flow under various weather conditions. Compared with the optical flow of clean scene, rain brings many additional artifacts in the optical flow while fog smoothens the entire optical flow. This motivates us to categorize the impact of adverse weather into dynamic weather and static weather for different motion adaptation strategies.
• Figure 5: Analysis of static weather (e.g., fog) at different depths. The deeper the depth, the more severe the image, and the more degraded the optical flow. Depth is the key to static weather, motivating us to build a depth association strategy for bridging the clean-degraded knowledge transfer.