Towards Robust Image Stitching: An Adaptive Resistance Learning against Compatible Attacks

TL;DR

This work tackles the vulnerability of learning-based image stitching to imperceptible adversarial perturbations that disrupt feature matching and alignment. It introduces a stitching-oriented attack (SoA) to maximize alignment loss and an adaptive adversarial training (AAT) framework based on differentiable architecture search to build robust stitching models. The robust model is built on a PWCnet backbone with a three-scale pyramid and communal cells, optimized to balance attack resistance and stitching accuracy; SoA degrades traditional models while AAT achieves superior robustness with minimal quality loss. Empirical results on synthetic MS-COCO-based and real-world UDIS-D datasets show SoA effectively challenges stitching methods, and AAT consistently outperforms routine adversarial training, delivering reliable stitching under attack with code available at the authors’ repository.

Abstract

Image stitching seamlessly integrates images captured from varying perspectives into a single wide field-of-view image. Such integration not only broadens the captured scene but also augments holistic perception in computer vision applications. Given a pair of captured images, subtle perturbations and distortions which go unnoticed by the human visual system tend to attack the correspondence matching, impairing the performance of image stitching algorithms. In light of this challenge, this paper presents the first attempt to improve the robustness of image stitching against adversarial attacks. Specifically, we introduce a stitching-oriented attack~(SoA), tailored to amplify the alignment loss within overlapping regions, thereby targeting the feature matching procedure. To establish an attack resistant model, we delve into the robustness of stitching architecture and develop an adaptive adversarial training~(AAT) to balance attack resistance with stitching precision. In this way, we relieve the gap between the routine adversarial training and benign models, ensuring resilience without quality compromise. Comprehensive evaluation across real-world and synthetic datasets validate the deterioration of SoA on stitching performance. Furthermore, AAT emerges as a more robust solution against adversarial perturbations, delivering superior stitching results. Code is available at:https://github.com/Jzy2017/TRIS.

Figures (9)

• Figure 1: Illustration of our motivation. (a) presents the reference performance of prolific RSFI nie2021unsupervised on benign images. (b) reveals the vulnerability of RSFI under adversarial perturbations. Upon routine adversarial training, the robustness of RSFI improves, yet there remains a notable performance decline, as depicted in (c). In contrast, the proposed method in (d) not only exhibits resilience against perturbations but also delivers a performance surpassing that observed in the benign scenarios.
• Figure 2: Illustration of the stitching-oriented attacks (SoA) based routine adversarial training (a) and the proposed adaptive adversarial training (AAT) (b).The basic architecture we employed is shown in (c).
• Figure 3: Results of our method under different attacks (i.e., FGSM, BIM, PGD and SoA ) and in attack-free (benign) scenarios.
• Figure 4: Visual comparisons of different adversarially trained stitching models on benign and attacked images.
• Figure 5: Performance deterioration from different attacks on deep learning models. Blue, yellow and red denote VFIS, RSFI and our baseline model trained with clean data.