Learning Data Association for Multi-Object Tracking using Only Coordinates

TL;DR

A novel Transformer-based module, named TWiX, is trained on sets of tracks with the objective of discriminating pairs of tracks coming from the same object from those which are not, and achieves state-of-the-art performance on the DanceTrack and KITTIMOT datasets and gets competitive results on the MOT17 dataset.

Abstract

We propose a novel Transformer-based module to address the data association problem for multi-object tracking. From detections obtained by a pretrained detector, this module uses only coordinates from bounding boxes to estimate an affinity score between pairs of tracks extracted from two distinct temporal windows. This module, named TWiX, is trained on sets of tracks with the objective of discriminating pairs of tracks coming from the same object from those which are not. Our module does not use the intersection over union measure, nor does it requires any motion priors or any camera motion compensation technique. By inserting TWiX within an online cascade matching pipeline, our tracker C-TWiX achieves state-of-the-art performance on the DanceTrack and KITTIMOT datasets, and gets competitive results on the MOT17 dataset. The code will be made available upon publication.

Figures (7)

• Figure 1: Creation of a batch of tracklets with two temporal windows used during the training. The two frames of reference are $f_P$=4 and $f_F$=6 and the temporal windows are of length $t_P$=3 and $t_F$=2 frames. a) The set of past and future tracklets contains each four tracklets. Gray detections are completely ignored in this batch of tracklets. b) The matrix $\textbf{Y}$ indicates whether a pair is positive (1), negative (0) or ignored (?). Best viewed in color.
• Figure 2: Architecture of TWiX (read from bottom to top). First, pairs of tracklets are normalized and linearly projected then encoded with a Transformer where attention is applied on the temporal dimension. Then, refined representations are obtained with a second Transformer which pays attention to all other pairs. Finally, a linear layer and a hyperbolic tangent function are used to compute an affinity score for each pair. Best viewed in color.
• Figure 3: Our tracker C-TWiX use a cascade matching pipeline for tracking. The BIoU-computed matrix in C-BIoU is replaced by our TWiX module. Best viewed in color.
• Figure 4: Comparison of HOTA on the validation set of KITTIMOT-car at different level of matching regarding the presence of the Inter-Pair Transformer Encoder (left) or not (right).
• Figure 5: HOTA scores on KITTIMOT and MOT17 validation sets with regard to the loss function. Red and blue indicate respectively the first and second best methods.