Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Track Anything Rapter(TAR)

Tharun V. Puthanveettil, Fnu Obaid ur Rahman

TL;DR

Track Anything Rapter (TAR) presents a UAV-based system that detects, segments, and tracks arbitrary targets using multimodal queries by integrating Segment Anything Model (SAM), DINO, and CLIP within a visual servoing framework. It addresses open-set adaptability and intuitive user interaction by enabling object specification through bounding boxes, image templates, or clicks, paired with a proportional high-level controller implemented in ROS2/MAVROS on a PX4-enabled drone. The evaluation, combining ROS2 Gazebo simulations and real hardware with Vicon ground truth, demonstrates robust tracking under occlusion and across modalities, with DTW-based trajectory comparison showing DINO-based tracking achieving the best alignment (lowest mean DTW). Core contributions include a bespoke ROS2-based evaluation pipeline, a ground-truth–backed control loop, and DTW-based trajectory assessment, underscoring TAR's potential for flexible, real-time aerial tracking in dynamic environments.

Abstract

Object tracking is a fundamental task in computer vision with broad practical applications across various domains, including traffic monitoring, robotics, and autonomous vehicle tracking. In this project, we aim to develop a sophisticated aerial vehicle system known as Track Anything Rapter (TAR), designed to detect, segment, and track objects of interest based on user-provided multimodal queries, such as text, images, and clicks. TAR utilizes cutting-edge pre-trained models like DINO, CLIP, and SAM to estimate the relative pose of the queried object. The tracking problem is approached as a Visual Servoing task, enabling the UAV to consistently focus on the object through advanced motion planning and control algorithms. We showcase how the integration of these foundational models with a custom high-level control algorithm results in a highly stable and precise tracking system deployed on a custom-built PX4 Autopilot-enabled Voxl2 M500 drone. To validate the tracking algorithm's performance, we compare it against Vicon-based ground truth. Additionally, we evaluate the reliability of the foundational models in aiding tracking in scenarios involving occlusions. Finally, we test and validate the model's ability to work seamlessly with multiple modalities, such as click, bounding box, and image templates.

Track Anything Rapter(TAR)

TL;DR

Track Anything Rapter (TAR) presents a UAV-based system that detects, segments, and tracks arbitrary targets using multimodal queries by integrating Segment Anything Model (SAM), DINO, and CLIP within a visual servoing framework. It addresses open-set adaptability and intuitive user interaction by enabling object specification through bounding boxes, image templates, or clicks, paired with a proportional high-level controller implemented in ROS2/MAVROS on a PX4-enabled drone. The evaluation, combining ROS2 Gazebo simulations and real hardware with Vicon ground truth, demonstrates robust tracking under occlusion and across modalities, with DTW-based trajectory comparison showing DINO-based tracking achieving the best alignment (lowest mean DTW). Core contributions include a bespoke ROS2-based evaluation pipeline, a ground-truth–backed control loop, and DTW-based trajectory assessment, underscoring TAR's potential for flexible, real-time aerial tracking in dynamic environments.

Abstract

Object tracking is a fundamental task in computer vision with broad practical applications across various domains, including traffic monitoring, robotics, and autonomous vehicle tracking. In this project, we aim to develop a sophisticated aerial vehicle system known as Track Anything Rapter (TAR), designed to detect, segment, and track objects of interest based on user-provided multimodal queries, such as text, images, and clicks. TAR utilizes cutting-edge pre-trained models like DINO, CLIP, and SAM to estimate the relative pose of the queried object. The tracking problem is approached as a Visual Servoing task, enabling the UAV to consistently focus on the object through advanced motion planning and control algorithms. We showcase how the integration of these foundational models with a custom high-level control algorithm results in a highly stable and precise tracking system deployed on a custom-built PX4 Autopilot-enabled Voxl2 M500 drone. To validate the tracking algorithm's performance, we compare it against Vicon-based ground truth. Additionally, we evaluate the reliability of the foundational models in aiding tracking in scenarios involving occlusions. Finally, we test and validate the model's ability to work seamlessly with multiple modalities, such as click, bounding box, and image templates.
Paper Structure (19 sections, 10 equations, 13 figures, 1 table)

This paper contains 19 sections, 10 equations, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contribution
  3. Methodology
  4. Detection
  5. Re-detecting a lost object
  6. Experiment
  7. Hardware
  8. Simulation
  9. Implementation Details
  10. Input Source
  11. Process Flow
  12. Results
  13. Evaluation Metric
  14. Multimodal Input Queries
  15. Comparative Analysis: VOXL2 Apriltag Detection vs. FAN with Bounding Boxes vs FAN with DINO
  16. ...and 4 more sections

Figures (13)

  • Figure 1: Experimental setup used for testing the performance of the one-shot model-DINO in aiding in redetection of targets in case of obstructions.
  • Figure 3: Redesigned input pipeline for publishing the camera feed as an RTSP video stream.
  • Figure 4: Process Flow of the Project TAR
  • Figure 5: Image template based One-shot tracking pipeline using DINO
  • Figure 6: Oneshot Apriltag tracking based on the real-time bounding box query.
  • ...and 8 more figures