BankTweak: Adversarial Attack against Multi-Object Trackers by Manipulating Feature Banks

TL;DR

BankTweak tackles adversarial vulnerabilities in multi-object trackers by targeting the feature extractor in the association phase rather than object positions. By injecting crafted features into object banks and exploiting a vulnerability in the Hungarian matching step, it achieves persistent ID switches that endure after the attack ends, all without generating false positives. The approach is demonstrated across DeepSORT, StrongSORT, and MOTDT with diverse detectors, showing stronger IDF1 degradation and robustness to larger matching boundaries than prior attacks on MOT17 and MOT20. This work highlights a practical security concern for tracking-by-detection systems and motivates defense strategies focused on feature-bank integrity and matching robustness.

Abstract

Multi-object tracking (MOT) aims to construct moving trajectories for objects, and modern multi-object trackers mainly utilize the tracking-by-detection methodology. Initial approaches to MOT attacks primarily aimed to degrade the detection quality of the frames under attack, thereby reducing accuracy only in those specific frames, highlighting a lack of \textit{efficiency}. To improve efficiency, recent advancements manipulate object positions to cause persistent identity (ID) switches during the association phase, even after the attack ends within a few frames. However, these position-manipulating attacks have inherent limitations, as they can be easily counteracted by adjusting distance-related parameters in the association phase, revealing a lack of \textit{robustness}. In this paper, we present \textsf{BankTweak}, a novel adversarial attack designed for MOT trackers, which features efficiency and robustness. \textsf{BankTweak} focuses on the feature extractor in the association phase and reveals vulnerability in the Hungarian matching method used by feature-based MOT systems. Exploiting the vulnerability, \textsf{BankTweak} induces persistent ID switches (addressing \textit{efficiency}) even after the attack ends by strategically injecting altered features into the feature banks without modifying object positions (addressing \textit{robustness}). To demonstrate the applicability, we apply \textsf{BankTweak} to three multi-object trackers (DeepSORT, StrongSORT, and MOTDT) with one-stage, two-stage, anchor-free, and transformer detectors. Extensive experiments on the MOT17 and MOT20 datasets show that our method substantially surpasses existing attacks, exposing the vulnerability of the tracking-by-detection framework to \textsf{BankTweak}.

Figures (7)

• Figure 1: (a) Comparison between existing adversarial attacks ①--④ and BankTweak ⑤, (b) the principles behind ID switch induction in Hijacking ③ and the F&F attack ④, and (c) the diagram presenting the features of adversarial attacks.
• Figure 2: Generating perturbed features by iterating ①--⑥ to induce ID switch between two objects in BankTweak: a focus on the feature extractor during the association phase.
• Figure 3: Overall process by which BankTweak induces an ID switch between a pair of objects Ⓐ and Ⓑ, across five frames from $\tilde{I}_{t+1}$ to $\tilde{I}_{t+5}$; desired generated features are injected into the feature bank in $\tilde{I}_{t+1}$--$\tilde{I}_{t+3}$ (Step 1), and a permanent ID switch incurs by exploiting vulnerability in Hungarian matching method in $\tilde{I}_{t+4}$--$\tilde{I}_{t+5}$ (Step 2).
• Figure 4: (a) Comparison of cosine distance for two cases involving the same and different object sources, and (b)--(c) two attack failure scenarios of BankTweak.
• Figure 5: (a)--(c) IDF1 scores across varying Mahalanobis distance thresholds and (d)--(f) false alarms for three distinct trackers.