SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation

TL;DR

The paper tackles the challenge of transferring daytime UAV trackers to nighttime scenarios by addressing limited, high-quality nighttime training data. It introduces SAM-DA, a framework that leverages the Segment Anything Model to swell target-domain training samples from each nighttime image in a one-to-many fashion, coupled with a tracking-oriented domain adaptation pipeline featuring a backbone, a Transformer-based bridging module, a tracker head, and a discriminator. Empirical results on DarkTrack2021 and the long-term NUT-L benchmark show that SAM-DA-Track substantially surpasses Baseline and other SOTA trackers, achieving better performance with fewer raw nighttime images and demonstrating data-efficient improvements. The approach reduces labeling costs, accelerates validation, and enhances practical deployment of nighttime UAV tracking systems, underscoring SAM’s utility for domain adaptation in challenging perception tasks.

Abstract

Domain adaptation (DA) has demonstrated significant promise for real-time nighttime unmanned aerial vehicle (UAV) tracking. However, the state-of-the-art (SOTA) DA still lacks the potential object with accurate pixel-level location and boundary to generate the high-quality target domain training sample. This key issue constrains the transfer learning of the real-time daytime SOTA trackers for challenging nighttime UAV tracking. Recently, the notable Segment Anything Model (SAM) has achieved a remarkable zero-shot generalization ability to discover abundant potential objects due to its huge data-driven training approach. To solve the aforementioned issue, this work proposes a novel SAM-powered DA framework for real-time nighttime UAV tracking, i.e., SAM-DA. Specifically, an innovative SAM-powered target domain training sample swelling is designed to determine enormous high-quality target domain training samples from every single raw nighttime image. This novel one-to-many generation significantly expands the high-quality target domain training sample for DA. Comprehensive experiments on extensive nighttime UAV videos prove the robustness and domain adaptability of SAM-DA for nighttime UAV tracking. Especially, compared to the SOTA DA, SAM-DA can achieve better performance with fewer raw nighttime images, i.e., the fewer-better training. This economized training approach facilitates the quick validation and deployment of algorithms for UAVs. The code is available at https://github.com/vision4robotics/SAM-DA.

Figures (8)

• Figure 1: Overall performance of SAM-DA-Track and state-of-the-art (SOTA) trackers on the proposed NUT-L. SAM-DA-Track represents the version of the base tracker SiamBAN 9157457, adopting SAM-DA for adaptation training. N, T, S, and B represent that the target domain training images of SAM-DA-Track are about 10.0%, 33.2%, 50.1%, and 100% of the entire NAT2021-$train$9879981, respectively. With SAM, the proposed SAM-DA-Track shows superior performance with only 10% of training images of Baseline UDAT 9879981.
• Figure 2: Illustration of the proposed SAM-DA for nighttime UAV tracking. The original nighttime image is from NAT2021-$train$. The source domain training sample is from GOT-10k 8922619. The SAM-powered target domain training sample swelling is employed to determine enormous high-quality target domain training samples from every single nighttime image. Note the source domain (daytime) training samples are manually collected with time-consuming and expensive annotation, while the target domain (nighttime) training samples are automatically generated with our time-saving and low-cost swelling.
• Figure 3: Visualization of the target domain training samples processed by the SAM-powered target domain training sample swelling. The nighttime raw images are from NAT2021-$train$. With the novel one-to-many generation, many objects, such as buses, cars, riders, and signals are utilized to generate the training samples from a single image, enhancing the utilization efficiency of the raw nighttime data.
• Figure 4: Visual comparison of confidence maps generated by the Baseline and the SAM-DA-Track. Target objects are marked by red boxes. The nighttime images are from the proposed NUT-L. The Baseline exhibits sub-optimal performance in tracking tasks conducted under low-light conditions, whereas the proposed SAM-DA-Track method demonstrates notable efficacy in such scenarios.
• Figure 5: Overall performance of SAM-DA-Track and SOTA trackers on DarkTrack2021 9696362 and the proposed NUT-L. SAM-DA-Track significantly surpasses the Baseline and other SOTA methods.