SMART-TRACK: A Novel Kalman Filter-Guided Sensor Fusion For Robust UAV Object Tracking in Dynamic Environments

TL;DR

This work tackles robust UAV tracking in dynamic environments with intermittent sensor measurements. It introduces SMART-TRACK, a framework that fuses Kalman Filter state estimates with depth/RGB detection by projecting the KF mean and covariance into the sensor frame to form 2D depth-image search regions for reacquiring measurements, thereby maintaining high-frequency, accurate tracking. The method combines a YOLOv8 detection pipeline with KF-guided measurement augmentation, achieving RMSE as low as $0.04\ \mathrm{m}$ in dynamic scenarios and significantly reducing errors during detector outages. Validated in a ROS2/Gazebo simulation with realistic UAV dynamics and depth sensing, SMART-TRACK demonstrates notable improvements over prior depth-based and CNN-based approaches, offering a scalable, open-source solution for robust autonomous UAV operations.

Abstract

In the field of sensor fusion and state estimation for object detection and localization, ensuring accurate tracking in dynamic environments poses significant challenges. Traditional methods like the Kalman Filter (KF) often fail when measurements are intermittent, leading to rapid divergence in state estimations. To address this, we introduce SMART (Sensor Measurement Augmentation and Reacquisition Tracker), a novel approach that leverages high-frequency state estimates from the KF to guide the search for new measurements, maintaining tracking continuity even when direct measurements falter. This is crucial for dynamic environments where traditional methods struggle. Our contributions include: 1) Versatile Measurement Augmentation Using KF Feedback: We implement a versatile measurement augmentation system that serves as a backup when primary object detectors fail intermittently. This system is adaptable to various sensors, demonstrated using depth cameras where KF's 3D predictions are projected into 2D depth image coordinates, integrating nonlinear covariance propagation techniques simplified to first-order approximations. 2) Open-source ROS2 Implementation: We provide an open-source ROS2 implementation of the SMART-TRACK framework, validated in a realistic simulation environment using Gazebo and ROS2, fostering broader adaptation and further research. Our results showcase significant enhancements in tracking stability, with estimation RMSE as low as 0.04 m during measurement disruptions, advancing the robustness of UAV tracking and expanding the potential for reliable autonomous UAV operations in complex scenarios. The implementation is available at https://github.com/mzahana/SMART-TRACK.

Figures (7)

• Figure 1: SMART-TRACK system flowchart
• Figure 2: Target expressed in color (RGB) and depth image frames.
• Figure 3: The target's position expressed in map frame (green line), camera 3D coordinate frame (red line), and the camera 2D image frame ($^{C_{2D}}X$)
• Figure 4: The search ellipse $E$ in the depth image $I_{depth}$. The original $E$ is constructed using the eigenvalues and eigenvectors of $^{C_{2D}}\bm{\Sigma}$ (green). The scaled ellipse $\bar{E}$ (red).
• Figure 5: KF error vs. time for the static target scenario, with and without SMART-TRACK.