Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow

TL;DR

This work addresses nighttime optical flow, where texture degradation and noise hinder reliable motion estimation. It introduces ABDA-Flow, which learns two intermediate common spaces—a reflectance-aligned appearance space bridging daytime and nighttime visuals via intrinsic image decomposition, and a spatiotemporal gradient-aligned boundary space linking nighttime frames with accumulated events—to transfer global motion and local boundary knowledge, respectively. Appearance adaptation leverages Retinex-based decomposition, adversarial reflectance alignment, and KL-divergence on cost volumes to transfer motion, while boundary adaptation uses a derived image-event correlation in the gradient space and contrastive learning to transfer boundary cues, with a joint optimization that combines eight loss terms into $L_{ABDA}$. Across synthetic and real nighttime datasets, ABDA-Flow set new state-of-the-art results by effectively mitigating illumination-induced domain gaps, demonstrating the value of a common-space adaptation paradigm for heterogeneous representations in optical flow and suggesting broader applicability to adverse-scene understanding.

Abstract

We investigate a challenging task of nighttime optical flow, which suffers from weakened texture and amplified noise. These degradations weaken discriminative visual features, thus causing invalid motion feature matching. Typically, existing methods employ domain adaptation to transfer knowledge from auxiliary domain to nighttime domain in either input visual space or output motion space. However, this direct adaptation is ineffective, since there exists a large domain gap due to the intrinsic heterogeneous nature of the feature representations between auxiliary and nighttime domains. To overcome this issue, we explore a common-latent space as the intermediate bridge to reinforce the feature alignment between auxiliary and nighttime domains. In this work, we exploit two auxiliary daytime and event domains, and propose a novel common appearance-boundary adaptation framework for nighttime optical flow. In appearance adaptation, we employ the intrinsic image decomposition to embed the auxiliary daytime image and the nighttime image into a reflectance-aligned common space. We discover that motion distributions of the two reflectance maps are very similar, benefiting us to consistently transfer motion appearance knowledge from daytime to nighttime domain. In boundary adaptation, we theoretically derive the motion correlation formula between nighttime image and accumulated events within a spatiotemporal gradient-aligned common space. We figure out that the correlation of the two spatiotemporal gradient maps shares significant discrepancy, benefitting us to contrastively transfer boundary knowledge from event to nighttime domain. Moreover, appearance adaptation and boundary adaptation are complementary to each other, since they could jointly transfer global motion and local boundary knowledge to the nighttime domain.

Figures (7)

• Figure 1: Illustration of three nighttime optical flow paradigms. (a) Visual adaptation. (b) Motion adaptation. (c) Common adaptation. Visual adaptation and motion adaptation methods directly transfer knowledge from source to target domain in input visual space and output motion space, respectively. However, this direct adaptation is ineffective due to the large domain gap caused by the intrinsic heterogeneous nature (feature distribution misalignment) of the feature representations between source and target domains. In contrast, we explore a common-latent space as the intermediate bridge to reinforce the feature alignment between source and target domains. In this work, we employ daytime domain and event domain as the source domains, and build the reflectance-aligned and spatiotemporal gradient-aligned common spaces to transfer knowledge to target nighttime domain.
• Figure 2: The architecture of the ABDA-Flow mainly contains appearance and boundary adaptation. In appearance adaptation, we take retinex model to align daytime and nighttime images into the reflectance-aligned common space. We then map the common features to motion space, and make the motion distributions between daytime and nighttime domains aligned. In boundary adaptation, we transform nighttime image and event stream to the spatiotemporal gradient-aligned common space. We then calculate the correlation statistic between the two spatiotemporal gradient maps to generate an attention map for guiding the boundary features alignment between nighttime and event domains.
• Figure 3: Motion distribution of daytime and nighttime domains. Optical flow of nighttime frame suffers degradation while flow of daytime frame is sharp. Motion distribution of nighttime reflectance is similar to those of daytime frame and reflectance, but dissimilar to that of nighttime frame. This motivates us to take reflectance as the common latent space to transfer knowledge.
• Figure 4: Motion correlation statistic between nighttime image and event domains. We use Euclidean distance to calculate the motion correlation between the nighttime image and accumulated events within the spatiotemporal gradient common space. The larger the distance is, the larger correlation discrepancy is, and the more dissimilar the boundaries of the two spatiotemporal gradient maps. This motivates us to contrastively transfer boundary knowledge to nighttime image domain.
• Figure 5: Visual comparison of optical flows on real nighttime images of DSEC dataset.