Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs

Jia Syuen Lim, Yadan Luo, Zhi Chen, Tianqi Wei, Scott Chapman, Zi Huang

TL;DR

This work leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention, and uses these pseudo-labels to train a YOLOv8 segmentation network.

Abstract

In the Detection and Multi-Object Tracking of Sweet Peppers Challenge, we present Track Any Peppers (TAP) - a weakly supervised ensemble technique for sweet peppers tracking. TAP leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention. These pseudo-labels, refined when necessary, are used to train a YOLOv8 segmentation network. To enhance detection accuracy under challenging conditions, we incorporate pre-processing techniques such as relighting adjustments and apply depth-based filtering during post-inference. For object tracking, we integrate the Matching by Segment Anything (MASA) adapter with the BoT-SORT algorithm. Our approach achieves a HOTA score of 80.4%, MOTA of 66.1%, Recall of 74.0%, and Precision of 90.7%, demonstrating effective tracking of sweet peppers without extensive manual effort. This work highlights the potential of foundation models for efficient and accurate object detection and tracking in agricultural settings.

Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs

TL;DR

This work leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention, and uses these pseudo-labels to train a YOLOv8 segmentation network.

Abstract

In the Detection and Multi-Object Tracking of Sweet Peppers Challenge, we present Track Any Peppers (TAP) - a weakly supervised ensemble technique for sweet peppers tracking. TAP leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention. These pseudo-labels, refined when necessary, are used to train a YOLOv8 segmentation network. To enhance detection accuracy under challenging conditions, we incorporate pre-processing techniques such as relighting adjustments and apply depth-based filtering during post-inference. For object tracking, we integrate the Matching by Segment Anything (MASA) adapter with the BoT-SORT algorithm. Our approach achieves a HOTA score of 80.4%, MOTA of 66.1%, Recall of 74.0%, and Precision of 90.7%, demonstrating effective tracking of sweet peppers without extensive manual effort. This work highlights the potential of foundation models for efficient and accurate object detection and tracking in agricultural settings.

Paper Structure

This paper contains 10 sections, 3 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation.
  4. Weak Labels Acquisition with Foundation Models.
  5. Mask & Box Detection with YOLOv8.
  6. Adaptive Relighting Scheme.
  7. Detection and Depth-Based Filtering.
  8. Hybrid Object Tracking with Ensemble Methods.
  9. Experiments
  10. Conclusion

Figures (1)

  • Figure 1: The overall framework of TAP. First, pseudo-labels are generated using a foundation model and refined by experts to ensure quality. SAM is then used to create segmentation masks based on these refined labels. The labels are subsequently employed to train a segmentation model for sweet pepper detection. Before inference, adaptive relighting compensates for illumination variance, while post-inference depth filtering separates object layers. Finally, a hybrid tracking system combines algorithms to assign unique tracking IDs and ensure consistency across frames.