RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo

TL;DR

RobustSpring tackles the lack of robustness evaluation in dense matching by introducing a corruption-focused benchmark for optical flow, scene flow, and stereo. It builds on the Spring dataset by applying 20 image corruptions in time, stereo, and depth, creating 40,000 frames and enabling two-axis evaluation of accuracy and robustness through a Lipschitz-based metric $L^c = \frac{\|f(I)-f(I^c)\|}{\|I-I^c\|}$ and corruption-specific scores $R^c_M$. The contributions include (i) a comprehensive corruption dataset with 16 depth/stereo/time-consistent corruptions, (ii) a ground-truth-free robustness metric $R^c_M$ accompanied by per-corruption variants like $R^c_{EPE}$, $R^c_{1px}$ and $R^c_{Fl}$, and (iii) public benchmarking tightly integrated with Spring to enable two-axis comparisons of model accuracy and robustness. Empirical results across 16 models reveal that high accuracy does not guarantee robustness and that robustness varies markedly by corruption type; transformer-based architectures usually excel in overall robustness but show weaknesses under certain corruptions. By providing data-efficient evaluation and transferability to real-world data (KITTI), RobustSpring offers a practical path toward designing and selecting dense-matching models that maintain reliable performance under real-world image degradations.

Abstract

Standard benchmarks for optical flow, scene flow, and stereo vision algorithms generally focus on model accuracy rather than robustness to image corruptions like noise or rain. Hence, the resilience of models to such real-world perturbations is largely unquantified. To address this, we present RobustSpring, a comprehensive dataset and benchmark for evaluating robustness to image corruptions for optical flow, scene flow, and stereo models. RobustSpring applies 20 different image corruptions, including noise, blur, color changes, quality degradations, and weather distortions, in a time-, stereo-, and depth-consistent manner to the high-resolution Spring dataset, creating a suite of 20,000 corrupted images that reflect challenging conditions. RobustSpring enables comparisons of model robustness via a new corruption robustness metric. Integration with the Spring benchmark enables public two-axis evaluations of both accuracy and robustness. We benchmark a curated selection of initial models, observing that accurate models are not necessarily robust and that robustness varies widely by corruption type. RobustSpring is a new computer vision benchmark that treats robustness as a first-class citizen to foster models that combine accuracy with resilience. It will be available at https://spring-benchmark.org.

Figures (11)

• Figure 1: RobustSpring is a novel image corruption benchmark for optical flow, scene flow and stereo. It evaluates 20 image corruptions including blurs, color changes, noises, quality degradations, and weather, applied to stereo video data from Mehl2023SpringHighResolution. For comprehensive robustness evaluations on all three tasks, RobustSpring's image corruptions are integrated in time, stereo and depth where applicable.
• Figure 2: RobustSpring's image corruptions on a single image.
• Figure 3: RobustSpring example frames. The first row shows clean and corrupted images. The second row shows the left and right disparity maps predicted with LEA Stereo Cheng2020_LEAStereo. The third row shows the target disparities for forward left, backward left, forward right, and backward right directions from M-FUSE Mehl2023_MFUSE. The fourth row shows optical flow estimates for forward left, backward left, forward right, and backward right from RAFT teed2020raft. All disparities and flows are computed on the corrupted dataset, see \ref{['fig:examples_1']} in the Supplementary for additional frames.
• Figure 4: Accumulated corruption robustness $R^c_\text{EPE}$ for optical flow models over all corruptions [top] and only noise corruptions [bottom]. All other corruption classes color (purple), blur (blue), noise (cyan), quality (green), and weather (yellow) are in \ref{['fig:relative_epe_by_group_supp']}.
• Figure 5: Accuracy vs. robustness of optical flow models, measured as EPE and Averaged $R^c_\text{EPE}$. Small values indicate accurate and robust models. \ref{['fig:accuracy_vs_robustness_median']} shows accuracy vs. Median $R^c_\text{EPE}$.