Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Halima Bouzidi, Haoyu Liu, Yonatan Gizachew Achamyeleh, Praneetsai Vasu Iddamsetty, Mohammad Abdullah Al Faruque

Abstract

Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.

Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

Abstract

Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.

Paper Structure

This paper contains 64 sections, 41 equations, 12 figures, 13 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-object Tracking (MOT).
  4. Adversarial Attacks on MOT
  5. Physical Adversarial Examples for MOT
  6. FADE: Adversarial Attack Design
  7. A Primer on TBP-based Trackers
  8. Query Set Initialization at Frame $F_t$
  9. Deformable Transformer Decoder
  10. Prediction and Track Query Update
  11. Adversarial Attacks Formulation
  12. Temporal Query Flooding (TQF)
  13. Temporal Memory Corruption (TMC)
  14. Digital-to-Physical Adversarial Examples
  15. Gradient-based Attack Optimization
  16. ...and 49 more sections

Figures (12)

  • Figure 1: Overview of the Proposed FADE Attack Strategies.
  • Figure 2: Overview of the FADE Attack Pipeline. (a) A PGD differentiable optimization loop crafts digital or physical perturbations $\omega$. (b) The TBP tracker ($f_{MOT}$) processes the perturbation, and its outputs/hidden-states ($\hat{Y}^t$ and $\mathcal{H}^t$) are used to calculate (bottom) $\mathcal{L}_{\text{FADE}} \in \{\mathcal{L}_{\text{TQF}} , \mathcal{L}_{\text{TMC}}\}$. (c) The visualized results show $\mathcal{L}_{\text{TQF}}$ causing query starvation via spurious track injection and $\mathcal{L}_{\text{TMC}}$ causing re-association failure via temporal memory corruption. The loss gradient is fed back to the optimization loop (a), closing the FADE pipeline.
  • Figure 3: Visualization of the TQF attack effectiveness against MeMOTR gao2023memotr on a video sequence from the MOT17 benchmark.
  • Figure 4: Qualitative comparison of the TMC attack with AAI (upper) and EAI (bottom) attack vectors on the MOT17 benchmark.
  • Figure 5: Physical Adversarial Attacks Scenarios and Models.
  • ...and 7 more figures