DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions

TL;DR

DispBench introduces the first robustness benchmark for disparity estimation, addressing the critical need for reliable performance under distribution shifts and adversarial perturbations in safety-critical applications. It systematically evaluates i.i.d. accuracy, generalization to 2D Common Corruptions, and reliability under white-box attacks across FlyingThings3D and KITTI2015 using a model zoo that includes GWCNet, CFNet, STTR, and STTR-light. Key findings reveal that improvements in i.i.d. performance do not guarantee robustness or generalization, with weather-like corruptions posing the greatest challenge and synthetic corruptions on synthetic data failing to predict real-world behavior. The work highlights the inadequacy of synthetic proxies and argues for real-world corruption evaluations, while outlining future integration with broader ecosystems like OpenStereo and addressing computational limitations to broaden coverage.

Abstract

Deep learning (DL) has surpassed human performance on standard benchmarks, driving its widespread adoption in computer vision tasks. One such task is disparity estimation, estimating the disparity between matching pixels in stereo image pairs, which is crucial for safety-critical applications like medical surgeries and autonomous navigation. However, DL-based disparity estimation methods are highly susceptible to distribution shifts and adversarial attacks, raising concerns about their reliability and generalization. Despite these concerns, a standardized benchmark for evaluating the robustness of disparity estimation methods remains absent, hindering progress in the field. To address this gap, we introduce DispBench, a comprehensive benchmarking tool for systematically assessing the reliability of disparity estimation methods. DispBench evaluates robustness against synthetic image corruptions such as adversarial attacks and out-of-distribution shifts caused by 2D Common Corruptions across multiple datasets and diverse corruption scenarios. We conduct the most extensive performance and robustness analysis of disparity estimation methods to date, uncovering key correlations between accuracy, reliability, and generalization. Open-source code for DispBench: https://github.com/shashankskagnihotri/benchmarking_robustness/tree/disparity_estimation/final/disparity_estimation

Figures (9)

• Figure 1: Analyzing the generalization ability of some Disparity estimation methods: GWCNet gwcnet, CFNet cfnet, and STTR and STTR-light sttr proposed over time. The y-axis represents the mean End-Point-Error (EPE) on Syntheticc Corruptions (2D Common Corrruptions commoncorruptions) at different severalties (severity=0 is i.i.d. performance) using the FlyingThings3D flyingthings_dispnet, i.e., lower is better. We observe that disparity estimation methods lack the generalization ability to common corruptions and, thus, are not safe for real-world deployment.
• Figure 2: Example of performing adversarial attacks on STTR using KITTI2015 dataset under different attacks. We show the samples before and after the attacks and the predictions before and after the respective adversarial attacks.
• Figure 3: Example of predictions using STTR on KITTI2015 dataset under different severities of the 2D Common Corruption: Frost.
• Figure 4: Using the FlyingThings3D dataset for disparity estimation, we perform an initial benchmarking of i.i.d. performance and generalization abilities of four popular disparity estimation methods. CFNet and GWCNet are traditional CNN-based stereo matching methods, whereas STTR and STTR-light are newly proposed transformer-based large models capable of zero-shot disparity estimation. Here, we use their fine-tuned versions for the FlyingThings3D dataset. The y-axis reports the mean EPE over the entire validation set for the respective corruption, and the x-axis denotes the severity of the 2D Common Corruption used to corrupt the input images. We report the i.i.d. performance at severity=0. Here we observe that while all four methods are highly vulnerable to Noise and Weather corruptions, newly proposed STTR and STTR-light are surprisingly less robust than the older CNN-based methods against weather corruptions. This finding is interesting and concerning as weather corruptions are the most likely real-world domain shift.
• Figure 5: Using the KITTI2015 dataset for disparity estimation, we perform an initial benchmarking of i.i.d. performance and generalization abilities of the two popular and available disparity estimation methods. GWCNet is a traditional CNN-based stereo matching method, whereas STTR is a newly proposed transformer-based large model capable of zero-shot disparity estimation. Here, we use their fine-tuned versions for the KITTI2015 dataset. The y-axis reports the mean EPE over the entire validation set for the respective corruption, and the x-axis denotes the severity of the 2D Common Corruption used to corrupt the input images. We report the i.i.d. performance at severity=0. Here, we observe that while both the methods are highly vulnerable to Noise and Weather corruptions, the newly proposed STTR is surprisingly less robust than the older CNN-based method against all corruptions. This finding is interesting and concerning as it contradicts the findings on the Synthetic Dataset FlyingThings3D in \ref{['fig:2dcc_perf_sceneflow']}.