FlyCo: Foundation Model-Empowered Drones for Autonomous 3D Structure Scanning in Open-World Environments

TL;DR

FlyCo tackles autonomous, prompt-driven 3D structure scanning in open-world environments by integrating foundation-model knowledge into a perception–prediction–planning loop. The system grounds high-level prompts into robust 2D/3D target segmentation, completes partially observed geometry with a multi-modal FM predictor, and guides efficient, safe long-horizon flight via a prediction-aware hierarchical planner. Key contributions include (i) a prompt-grounded perception module with streaming SAM2 adaptation and cross-modal refinement, (ii) a multi-modal surface predictor that fuses geometry, vision, and language with alternating attention and automatic data synthesis, and (iii) a hierarchical planner that couples global, consistency-aware coverage with fast local trajectory optimization. Real-world demonstrations and large-scale simulations show FlyCo reduces flight time by 1.25×–3.0× and increases target coverage by 4.3–56.2 percentage points while maintaining high safety and reliability, validating the practicality of FM-guided aerial scanning. The work also provides ablations confirming the value of each module and outlines a blueprint for extending FM-based robotic perception and planning to broader open-world tasks.

Abstract

Autonomous 3D scanning of open-world target structures via drones remains challenging despite broad applications. Existing paradigms rely on restrictive assumptions or effortful human priors, limiting practicality, efficiency, and adaptability. Recent foundation models (FMs) offer great potential to bridge this gap. This paper investigates a critical research problem: What system architecture can effectively integrate FM knowledge for this task? We answer it with FlyCo, a principled FM-empowered perception-prediction-planning loop enabling fully autonomous, prompt-driven 3D target scanning in diverse unknown open-world environments. FlyCo directly translates low-effort human prompts (text, visual annotations) into precise adaptive scanning flights via three coordinated stages: (1) perception fuses streaming sensor data with vision-language FMs for robust target grounding and tracking; (2) prediction distills FM knowledge and combines multi-modal cues to infer the partially observed target's complete geometry; (3) planning leverages predictive foresight to generate efficient and safe paths with comprehensive target coverage. Building on this, we further design key components to boost open-world target grounding efficiency and robustness, enhance prediction quality in terms of shape accuracy, zero-shot generalization, and temporal stability, and balance long-horizon flight efficiency with real-time computability and online collision avoidance. Extensive challenging real-world and simulation experiments show FlyCo delivers precise scene understanding, high efficiency, and real-time safety, outperforming existing paradigms with lower human effort and verifying the proposed architecture's practicality. Comprehensive ablations validate each component's contribution. FlyCo also serves as a flexible, extensible blueprint, readily leveraging future FM and robotics advances. Code will be released.

Figures (24)

• Figure 1: Teaser. (A) Overview of the proposed FM-empowered aerial system to autonomously scan user-specified structures. It seamlessly integrates physical world understanding and knowledge built into FMs with advanced autonomous flight capabilities, translating low-effort human prompts (text descriptions, visual annotations) into precise drone actions for efficient target scanning in unknown open-world environments. This is achieved via a coordinated perception-prediction-planning loop, which operates entirely without human intervention or prior environmental knowledge. (B-C) Existing paradigms typically rely on effortful human priors and involvement, and tend to trigger re-flights for incomplete scans, both of which limit practicality, efficiency, and adaptability in open-world scenarios.
• Figure 2: In-the-wild demonstrations across four scenarios. The user-specified targets include: (A) an arch bridge linking buildings, (B) a large concert hall, (C) a castle gate, and (D) a red-brick building.
• Figure 3: System architecture of FlyCo. Given a human prompt, the system runs a closed-loop pipeline: leveraging FMs, perception grounds the target in 2D/3D from streaming RGB-LiDAR, and multi-modal prediction completes the target surface from partial observations; the planner asynchronously performs consistency-aware global coverage planning over the predicted geometry with real-time local trajectory replanning. All modules are continuously updated until the mission terminates.
• Figure 4: Overview of our multi-modal surface predictor. It first independently encodes each modality with foundation models, fuses heterogeneous features via alternating attention, and decodes the completed shape supervised by joint partial and completion Chamfer losses.
• Figure 5: Automatic multi-modal data generation pipeline, which synthesizes partial point clouds, RGB images, and textual descriptions from the original 3D models.