Towards Adaptive Meta-Gradient Adversarial Examples for Visual Tracking

TL;DR

The paper tackles security vulnerabilities of modern visual trackers to adversarial perturbations by introducing AMGA, an adaptive meta-gradient attack that fuses meta-learning with a multi-model ensemble. By formulating an ensemble probability $p_{ensemble}(x)=\sum_{i=1}^n \beta_i p_i(x)$ and optimizing perturbations through a two-stage meta-training/meta-testing loop, augmented with momentum $m_{k+1}=\mu m_k+\frac{\nabla_\delta \mathcal{L}_{ensemble}}{\|\nabla_\delta \mathcal{L}_{ensemble}\|}$ and Gaussian smoothing $\tilde{\delta}=\delta * \mathcal{G}_\sigma$, AMGA achieves strong transferability across unseen trackers without internal gradient access. The method is validated on seven trackers (SiamCAR, SiamRPN++, DiMP, MixFormer, TransT, OSTrack, SeqTrack) over OTB2015, GOT-10k, and LaSOT, showing significant degradation in precision and success rate compared with random noise and several black-box baselines, while remaining competitive with white-box attacks. This work advances understanding of tracker vulnerabilities and offers a practical framework for evaluating and benchmarking defenses, with implications for robust tracker design and adversarial defense research. Key formulations include $p_{ensemble}(x)=\sum_{i=1}^n \beta_i p_i(x)$, $\delta_{k+1}=\delta_k+\alpha\,\text{sign}(\nabla_\delta \mathcal{L}_{ensemble}(x+\delta_k,y))$, $m_{k+1}=\mu m_k+\frac{\nabla_\delta \mathcal{L}_{ensemble}(x+\delta_k,y)}{\|\nabla_\delta \mathcal{L}_{ensemble}(x+\delta_k,y)\|}$, and $\tilde{\delta}=\delta*\mathcal{G}_\sigma$ with $\mathcal{G}_\sigma(u,v)=\frac{1}{2\pi\sigma^2}\exp(-\frac{u^2+v^2}{2\sigma^2})$.

Abstract

In recent years, visual tracking methods based on convolutional neural networks and Transformers have achieved remarkable performance and have been successfully applied in fields such as autonomous driving. However, the numerous security issues exposed by deep learning models have gradually affected the reliable application of visual tracking methods in real-world scenarios. Therefore, how to reveal the security vulnerabilities of existing visual trackers through effective adversarial attacks has become a critical problem that needs to be addressed. To this end, we propose an adaptive meta-gradient adversarial attack (AMGA) method for visual tracking. This method integrates multi-model ensembles and meta-learning strategies, combining momentum mechanisms and Gaussian smoothing, which can significantly enhance the transferability and attack effectiveness of adversarial examples. AMGA randomly selects models from a large model repository, constructs diverse tracking scenarios, and iteratively performs both white- and black-box adversarial attacks in each scenario, optimizing the gradient directions of each model. This paradigm minimizes the gap between white- and black-box adversarial attacks, thus achieving excellent attack performance in black-box scenarios. Extensive experimental results on large-scale datasets such as OTB2015, LaSOT, and GOT-10k demonstrate that AMGA significantly improves the attack performance, transferability, and deception of adversarial examples. Codes and data are available at https://github.com/pgao-lab/AMGA.

Figures (6)

• Figure 1: AMGA attack for visual tracking. SiamRPN++ accurately locates the target object in the original frame, as shown in (a). Our proposed AMGA attack reduces tracking effectiveness by adding almost imperceptible perturbations, as shown in (b).
• Figure 2: Overview pipeline of the proposed AMGA attack method. AMGA randomly selects $n+1$ models from the model repository in each iteration to construct a meta-learning task. Each task is divided into two stages: meta-training and meta-testing. In the meta-training stage, a gradient-based perturbation generation is conducted using the set of the first $n$ models, and this process is iterated $k$ times. In the meta-testing stage, the adversarial perturbation obtained from meta-training is applied to the $(n+1)$-th model to obtain the final adversarial perturbation and example.
• Figure 3: Visualization of predicted bounding boxes of SiamCARr10, SiamRPN++r9, DiMPr22, MixFormerr23, TransTr24, OSTrackr48, and SeqTrackr49 on original frames, random noised examples, and AMGA adversarial examples from the OTB2015 benchmark datasetr43. Ground-truth bounding boxes are also provided for comparison.
• Figure 4: Visualization of predicted bounding boxes of SiamCARr10, SiamRPN++r9, DiMPr22, MixFormerr23, TransTr24, OSTrackr48, and SeqTrackr49 on original frames, random noised examples, and AMGA adversarial examples from the LaSOT benchmark datasetr44. Ground-truth bounding boxes are also provided for comparison.
• Figure 5: Evaluation on 11 different tracking scenarios of the OTB2015 benchmark datasetr43.