iKUN: Speak to Trackers without Retraining

TL;DR

This paper tackles referring multi-object tracking by decoupling the referring component from the tracker through an insertable module, iKUN. It introduces a Knowledge Unification Module (KUM) to adapt visual features under textual guidance, a Neural Kalman Filter (NKF) to dynamically adjust process and observation noise, and a test-time similarity calibration to handle open-set long-tail descriptions, achieving substantial gains on Refer-KITTI and Refer-Dance. The approach is plug-and-play with off-the-shelf trackers, offering improved accuracy (e.g., higher HOTA, DetA, and AssA) and reduced training costs, along with a new Refer-Dance dataset to advance RMOT research.

Abstract

Referring multi-object tracking (RMOT) aims to track multiple objects based on input textual descriptions. Previous works realize it by simply integrating an extra textual module into the multi-object tracker. However, they typically need to retrain the entire framework and have difficulties in optimization. In this work, we propose an insertable Knowledge Unification Network, termed iKUN, to enable communication with off-the-shelf trackers in a plug-and-play manner. Concretely, a knowledge unification module (KUM) is designed to adaptively extract visual features based on textual guidance. Meanwhile, to improve the localization accuracy, we present a neural version of Kalman filter (NKF) to dynamically adjust process noise and observation noise based on the current motion status. Moreover, to address the problem of open-set long-tail distribution of textual descriptions, a test-time similarity calibration method is proposed to refine the confidence score with pseudo frequency. Extensive experiments on Refer-KITTI dataset verify the effectiveness of our framework. Finally, to speed up the development of RMOT, we also contribute a more challenging dataset, Refer-Dance, by extending public DanceTrack dataset with motion and dressing descriptions. The codes and dataset are available at https://github.com/dyhBUPT/iKUN.

Figures (6)

• Figure 1: Comparison among previous RMOT frameworks and ours. (a) Previous methods incorporate the referring module into the multi-object tracker, which need to retrain the overall framework. (b) Instead, our designed model iKUN can be directly plugged after an off-the-shelf tracker, in which the tracker is frozen while training.
• Figure 2: The motivation of KUM. Given a tracklet and a set of descriptions, (a) without the guidance from textual stream, the visual encoder is asked to output a single feature to match multiple textual features; (b) with textual guidance, the visual encoder can predict adaptive features for each description.
• Figure 3: The overall framework of iKUN. The visual stream first embeds the local object feature $f_{local}$ and global scene feature $f_{global}$, and then aggregates them using the knowledge unification module (KUM). A temporal model and a visual head are followed to generate the final visual feature $f_v$. Meanwhile, the textual stream encodes the textual feature $f_t$. Finally, a logit head is utilized to predict the similarity score between $f_v$ and $f_t$
• Figure 4: Three designs of knowledge unification module. The feature maps are shown as the shape of their tensors with batch size $B$. For clarity, the final spatial global average pooling operation is omitted here.
• Figure 5: The performance of TransRMOTwu2023referring and iKUN on Refer-Dance.