Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks

TL;DR

This work addresses the vulnerability of modern no-reference image/video quality metrics to adversarial attacks by introducing a robustness benchmark that evaluates 15 NR IQA/VQA metrics under diverse attack families. It combines six open datasets, automated attack pipelines, and distribution-alignment via $1$-D Neural Optimal Transport to compare metrics on a common footing, reporting multiple robustness scores including $Abs ext{ gain}$, $Rel ext{ gain}$, $R_{score}$, $W_{score}$, and $E_{score}$. Key findings show that iterative attacks are more effective than universal perturbations, with MANIQA, META-IQA, NIMA, RANK-IQA, and MDTVSFA among the most robust, while FPR is consistently vulnerable; the results underscore the need for robustness-aware evaluation in benchmarks and applications. The authors provide online access to the leaderboard and open-source code to foster safer metric development and benchmarking for real-world image/video processing tasks.

Abstract

Nowadays, neural-network-based image- and video-quality metrics perform better than traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. Nonetheless, the adversarial robustness of image-quality metrics is also an area worth researching. This paper analyses modern metrics' robustness to different adversarial attacks. We adapted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image- and video-quality metrics. Some metrics showed high resistance to adversarial attacks, which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts submissions of new metrics for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. The latest results can be found online: https://videoprocessing.ai/benchmarks/metrics-robustness.html.

Figures (4)

• Figure 1: Metrics' robustness score for UAP-based adversarial attacks and SSIM measured between original and attacked images. The results are averaged for all test datasets.
• Figure 2: Metrics' robustness score for iterative adversarial attacks and SSIM measured between original and attacked images. The results are averaged for all test datasets.
• Figure 3: Dependency of metrics' robustness score of SSIM loss for attacked images (all types of attacks).
• Figure 4: Mean robustness score of compared metrics versus SSIM averages for UAP-based and iterative attacks.