Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaptFly: Prompt-Guided Adaptation of Foundation Models for Low-Altitude UAV Networks

Jiao Chen, Haoyi Wang, Jianhua Tang, Junyi Wang

TL;DR

AdaptFly addresses distribution shift in low-altitude UAV semantic segmentation by introducing prompt-guided test-time adaptation that keeps backbones frozen. It uses two complementary prompts—token prompts for resource-limited UAVs and gradient-free sparse visual prompts optimized by CMA-ES for resource-massive UAVs—together with an activation-statistic drift detector and a cross-UAV knowledge pool for asynchronous fleet-wide sharing. The approach achieves significant gains over static models and prior TTA baselines on UAVid and VDD, with strong results in Cityscapes→VDD and Cityscapes→UAVid cross-domain evaluations and validated real-world deployments under varying weather. These findings demonstrate a practical, communication-efficient pathway to resilient, scalable perception for the emerging low-altitude economy.

Abstract

Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on robust semantic segmentation as a foundational enabler for distributed sensing-communication-control co-design across heterogeneous agents within the network. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAVs cannot run gradient-based test-time adaptation, while resource-massive UAVs adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAVs, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAVs, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.

AdaptFly: Prompt-Guided Adaptation of Foundation Models for Low-Altitude UAV Networks

TL;DR

AdaptFly addresses distribution shift in low-altitude UAV semantic segmentation by introducing prompt-guided test-time adaptation that keeps backbones frozen. It uses two complementary prompts—token prompts for resource-limited UAVs and gradient-free sparse visual prompts optimized by CMA-ES for resource-massive UAVs—together with an activation-statistic drift detector and a cross-UAV knowledge pool for asynchronous fleet-wide sharing. The approach achieves significant gains over static models and prior TTA baselines on UAVid and VDD, with strong results in Cityscapes→VDD and Cityscapes→UAVid cross-domain evaluations and validated real-world deployments under varying weather. These findings demonstrate a practical, communication-efficient pathway to resilient, scalable perception for the emerging low-altitude economy.

Abstract

Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on robust semantic segmentation as a foundational enabler for distributed sensing-communication-control co-design across heterogeneous agents within the network. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAVs cannot run gradient-based test-time adaptation, while resource-massive UAVs adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAVs, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAVs, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.

Paper Structure

This paper contains 35 sections, 14 equations, 9 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Semantic Segmentation in Low-Altitude UAV Networks
  4. Prompt Tuning for Vision Foundation Models
  5. Test-Time Adaptation
  6. Problem Statement and Motivation
  7. Problem Definition
  8. Motivation
  9. Prompt Representations
  10. Token Prompts
  11. Sparse Visual Prompts
  12. Prompt-Guided Adaptation for Heterogeneous UAV Networks
  13. Resource-Limited UAVs: Network-Assisted Token Prompt Retrieval
  14. Resource-Massive UAV: Sparse Prompt Optimization via CMA-ES
  15. Domain Shift Detection via Activation Statistics for Prompt Management
  16. ...and 20 more sections

Figures (9)

  • Figure 1: Heterogeneous UAV networks with collaborative adaptation. ① Both resource-limited and resource-massive UAVs face performance degradation. ② Collaborative assistance via shared prompts.
  • Figure 2: Semantic segmentation as the perception backbone of low-altitude UAV networks.
  • Figure 3: Token prompt retrieval pipeline for resource-limited UAVs.
  • Figure 4: CMA-ES driven online SVP optimization.
  • Figure 5: Example of semantic segmentation results on the VDD dataset. GT denotes the ground truth annotations. Distinct differences in segmentation accuracy among methods (TENT, AdaptFly-Limited, and AdaptFly-Massive) are highlighted with yellow boxes.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Remark 1