Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

YOLO11-JDE: Fast and Accurate Multi-Object Tracking with Self-Supervised Re-ID

Iñaki Erregue, Kamal Nasrollahi, Sergio Escalera

TL;DR

YOLO11-JDE tackles real-time multi-object tracking by integrating a self-supervised Re-ID branch into a lightweight YOLO11s framework to perform Joint Detection and Embedding. It uses Mosaic data augmentation and a triplet loss with hard positive and semi-hard negative mining to learn discriminative appearance embeddings without heavy identity labeling, while a custom data association fuses motion, appearance, and localization cues for robust online tracking. Ablation studies show the optimal Re-ID loss setup, embedding dimension, and training data choices, with results on MOT17 and MOT20 demonstrating competitive accuracy and superior FPS using fewer than 10M parameters. The approach emphasizes practical applicability, particularly in crowded scenes, and suggests future work on better decoupling Re-ID from detection and exploring stronger augmentations and multi-scale embeddings for further gains.

Abstract

We introduce YOLO11-JDE, a fast and accurate multi-object tracking (MOT) solution that combines real-time object detection with self-supervised Re-Identification (Re-ID). By incorporating a dedicated Re-ID branch into YOLO11s, our model performs Joint Detection and Embedding (JDE), generating appearance features for each detection. The Re-ID branch is trained in a fully self-supervised setting while simultaneously training for detection, eliminating the need for costly identity-labeled datasets. The triplet loss, with hard positive and semi-hard negative mining strategies, is used for learning discriminative embeddings. Data association is enhanced with a custom tracking implementation that successfully integrates motion, appearance, and location cues. YOLO11-JDE achieves competitive results on MOT17 and MOT20 benchmarks, surpassing existing JDE methods in terms of FPS and using up to ten times fewer parameters. Thus, making our method a highly attractive solution for real-world applications.

YOLO11-JDE: Fast and Accurate Multi-Object Tracking with Self-Supervised Re-ID

TL;DR

YOLO11-JDE tackles real-time multi-object tracking by integrating a self-supervised Re-ID branch into a lightweight YOLO11s framework to perform Joint Detection and Embedding. It uses Mosaic data augmentation and a triplet loss with hard positive and semi-hard negative mining to learn discriminative appearance embeddings without heavy identity labeling, while a custom data association fuses motion, appearance, and localization cues for robust online tracking. Ablation studies show the optimal Re-ID loss setup, embedding dimension, and training data choices, with results on MOT17 and MOT20 demonstrating competitive accuracy and superior FPS using fewer than 10M parameters. The approach emphasizes practical applicability, particularly in crowded scenes, and suggests future work on better decoupling Re-ID from detection and exploring stronger augmentations and multi-scale embeddings for further gains.

Abstract

We introduce YOLO11-JDE, a fast and accurate multi-object tracking (MOT) solution that combines real-time object detection with self-supervised Re-Identification (Re-ID). By incorporating a dedicated Re-ID branch into YOLO11s, our model performs Joint Detection and Embedding (JDE), generating appearance features for each detection. The Re-ID branch is trained in a fully self-supervised setting while simultaneously training for detection, eliminating the need for costly identity-labeled datasets. The triplet loss, with hard positive and semi-hard negative mining strategies, is used for learning discriminative embeddings. Data association is enhanced with a custom tracking implementation that successfully integrates motion, appearance, and location cues. YOLO11-JDE achieves competitive results on MOT17 and MOT20 benchmarks, surpassing existing JDE methods in terms of FPS and using up to ten times fewer parameters. Thus, making our method a highly attractive solution for real-world applications.
Paper Structure (20 sections, 1 equation, 4 figures, 8 tables)

This paper contains 20 sections, 1 equation, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tracking-by-Detection
  4. Re-Identification
  5. Joint Detection and Embedding
  6. YOLO11-JDE
  7. Architecture
  8. Self-Supervised Training Strategy
  9. Re-ID Loss
  10. Data Association
  11. Experiments
  12. Dataset and Metrics
  13. Implementation Details
  14. Ablative Studies
  15. Re-ID Loss
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Comparative analysis of JDE models on the MOT20 test set. MOTA and FPS on the vertical and horizontal axis respectively. ID Switches are represented by the bubble size. YOLO11-JDE achieves a strong balance between tracking performance and inference speed.
  • Figure 2: Comparison of tracking architectures: (a) Separate Detection and Embedding (SDE), where detection and Re-ID are performed by separate models; (b) Joint Detection and Embedding (JDE), integrating detection and Re-ID into a single model; (c) Basic YOLO model structure including the backbone, neck, and multiple output heads; and (d) YOLO11-JDE head, featuring a specialized Re-ID branch. Grey parallelograms represent data, while rounded rectangles depict models. Our main contributions are highlighted in orange.
  • Figure 3: Example of four training images using Mosaic data augmentation for JDE. This technique combines multiple images, showing several identities (e.g., IDs 45543, 45544, 45549) under diverse transformations in a single input image and/or batch.
  • Figure 4: Example of consistent identity maintenance across frames on the MOT20-04 sequence despite multiple occlusions (ID 248).