Benchmarking the Robustness of Optical Flow Estimation to Corruptions

TL;DR

The KITTI-FC and GoPro-FC robustness benchmark is established as the first corruption robustness benchmark for optical flow estimation, with Out-Of-Domain (OOD) and In-Domain (ID) settings to facilitate comprehensive studies.

Abstract

Optical flow estimation is extensively used in autonomous driving and video editing. While existing models demonstrate state-of-the-art performance across various benchmarks, the robustness of these methods has been infrequently investigated. Despite some research focusing on the robustness of optical flow models against adversarial attacks, there has been a lack of studies investigating their robustness to common corruptions. Taking into account the unique temporal characteristics of optical flow, we introduce 7 temporal corruptions specifically designed for benchmarking the robustness of optical flow models, in addition to 17 classical single-image corruptions, in which advanced PSF Blur simulation method is performed. Two robustness benchmarks, KITTI-FC and GoPro-FC, are subsequently established as the first corruption robustness benchmark for optical flow estimation, with Out-Of-Domain (OOD) and In-Domain (ID) settings to facilitate comprehensive studies. Robustness metrics, Corruption Robustness Error (CRE), Corruption Robustness Error ratio (CREr), and Relative Corruption Robustness Error (RCRE) are further introduced to quantify the optical flow estimation robustness. 29 model variants from 15 optical flow methods are evaluated, yielding 10 intriguing observations, such as 1) the absolute robustness of the model is heavily dependent on the estimation performance; 2) the corruptions that diminish local information are more serious than that reduce visual effects. We also give suggestions for the design and application of optical flow models. We anticipate that our benchmark will serve as a foundational resource for advancing research in robust optical flow estimation. The benchmarks and source code will be released at https://github.com/ZhonghuaYi/optical_flow_robustness_benchmark.

Figures (8)

• Figure 1: Effects of all the $24$ corruptions under severity of $5$. Corruptions are split into $6$ classes. $7$ temporal corruptions are in red boxes. The previous and next frames are displayed on the left and right sides of the image respectively for Over Exposure and Under Exposure for better visualization. Examples are from GoPro-FC.
• Figure 2: Calculating procedure of $\text{CRE}_{c,s}$ and $\text{RCRE}_{c,s}$.$\text{RCRE}_{c,s}$ is computed without using ground-truth optical flow.
• Figure 3: The EPE, CRE, and CREr results of $12$ optical flow models on OOD and ID benchmarks of KITTI-FC. CREr is represented by the size of the bubble and its value is indicated below the model name. Purple circles represent CNN-based models, green circles represent Transformer-based models, and blue circles represent unsupervised models.
• Figure 4: CREr of representative models on different corruption classes. Weather corruptions heavily influence the supervised models from OOD to ID.
• Figure 5: SAM Segmentation results on (a) KITTI-FC and (b) GoPro-FC. Small motions in GoPro-FC result the pixel-level segmentation misalignment not helpful for optical flow estimation.