Leveraging Foundation Models via Knowledge Distillation in Multi-Object Tracking: Distilling DINOv2 Features to FairMOT

TL;DR

This work investigates utilizing foundation-model representations via knowledge distillation to improve multi-object tracking (MOT). A teacher-student setup is used, with DINOv2 as the teacher and FairMOT (HRNetV2) as the student; a feature-distillation loss guides the student to mimic the teacher's embeddings, with a transform layer aligning patch-based DINOv2 outputs to spatial feature maps. Through extensive ablations, the study finds that cosine embedding loss, a simple single-layer feature transformation, and a balanced distillation weight ($\alpha=0.5$) with DINOv2 Base yield the best performance, though gains are dataset-dependent and often modest compared to the baseline FairMOT. The results reveal both the promise and the limitations of applying foundation models to MOT, highlighting the need for task-specific adaptation and careful architectural choices for practical impact.

Abstract

Multiple Object Tracking (MOT) is a computer vision task that has been employed in a variety of sectors. Some common limitations in MOT are varying object appearances, occlusions, or crowded scenes. To address these challenges, machine learning methods have been extensively deployed, leveraging large datasets, sophisticated models, and substantial computational resources. Due to practical limitations, access to the above is not always an option. However, with the recent release of foundation models by prominent AI companies, pretrained models have been trained on vast datasets and resources using state-of-the-art methods. This work tries to leverage one such foundation model, called DINOv2, through using knowledge distillation. The proposed method uses a teacher-student architecture, where DINOv2 is the teacher and the FairMOT backbone HRNetv2 W18 is the student. The results imply that although the proposed method shows improvements in certain scenarios, it does not consistently outperform the original FairMOT model. These findings highlight the potential and limitations of applying foundation models in knowledge

Figures (9)

• Figure 1: This figure displays the student-teacher architecture of the DINO model DINO.
• Figure 2: This figure shows the complete knowledge distillation pipeline. The data is fed in parallel to the DINOv2 model and the FairMOT model, both generating their features. The DINOv2 model returns the features from the last hidden layer. Similarly, the features of the FairMOT model are also from the last hidden layer, directly after the upsampling of the different convolution streams. Using distillation loss, the DINOv2 model teaches improved features to the FairMOT model.
• Figure 3: The architecture of the HRNet. As can be seen, the high resolution input stream is subdivided four times, into varying, from high to low, convolution streams. And at these divisions, information exchange through either aggregated strided convolutions or aggregated upsampled convolutions can be seen hrnet.
• Figure 4: This image displays the two proposed transformation methods. The top one is the simple single-layered transformation, while the one on the bottom is the complex multi-layered transformation.
• Figure 5: Example of how to structure the dataset files.