Physical ID-Transfer Attacks against Multi-Object Tracking via Adversarial Trajectory

TL;DR

AdvTraj exposes a real-time physical vulnerability in MOT by manipulating the ID-assignment stage through adversarial trajectories rather than perturbing detectors. The approach derives a differentiable objective to swap IDs between an attacker and a target, demonstrates complete success in white-box settings and strong transferability to multiple MOT models, and provides two human-executable maneuvers (Go-and-Stop, Stop-and-Go) to realize the attack in practice. The work highlights fundamental weaknesses in the association mechanism of tracking-by-detection MOT systems and discusses countermeasures, including depth fusion and robust motion-model training, while emphasizing the need for defenses that preserve benign tracking performance. Overall, AdvTraj demonstrates that ID integrity in MOT can be compromised online in physical spaces, motivating robustness enhancements and standardized evaluation of MOT system resilience to physical adversarial trajectories.

Abstract

Multi-Object Tracking (MOT) is a critical task in computer vision, with applications ranging from surveillance systems to autonomous driving. However, threats to MOT algorithms have yet been widely studied. In particular, incorrect association between the tracked objects and their assigned IDs can lead to severe consequences, such as wrong trajectory predictions. Previous attacks against MOT either focused on hijacking the trackers of individual objects, or manipulating the tracker IDs in MOT by attacking the integrated object detection (OD) module in the digital domain, which are model-specific, non-robust, and only able to affect specific samples in offline datasets. In this paper, we present AdvTraj, the first online and physical ID-manipulation attack against tracking-by-detection MOT, in which an attacker uses adversarial trajectories to transfer its ID to a targeted object to confuse the tracking system, without attacking OD. Our simulation results in CARLA show that AdvTraj can fool ID assignments with 100% success rate in various scenarios for white-box attacks against SORT, which also have high attack transferability (up to 93% attack success rate) against state-of-the-art (SOTA) MOT algorithms due to their common design principles. We characterize the patterns of trajectories generated by AdvTraj and propose two universal adversarial maneuvers that can be performed by a human walker/driver in daily scenarios. Our work reveals under-explored weaknesses in the object association phase of SOTA MOT systems, and provides insights into enhancing the robustness of such systems.

Figures (14)

• Figure 1: Illustration of potential consequences of ID-Transfer in surveillance and autonomous driving (AD) applications. In the surveillance scenario (above), ID-Transfer leads to wrong target of interest being tracked. In the AD scenario (below), ID-Transfer results in inaccurate trajectory prediction due to history trajectories that are inconsistent with ground truths.
• Figure 2: Illustration of tracking-by-detection pipeline in Multi-Object Tracking (MOT).
• Figure 3: Comparison of existing MOT attacks (by attacking OD modules in digital space) and our ID-Transfer attack (by adversarial trajectories in physical space).
• Figure 4: Illustration of AdvTraj's stages.
• Figure 5: Illustration of the two universal adversarial maneuvers for ID-Transfer. Top/lower walker is the attacker/non-cooperating target, respectively. Solid green boxes represent the detection results of the OD model, while dashed bounding boxes represent predictions made by the KF motion prediction module of the MOT system.