LongFly: Long-Horizon UAV Vision-and-Language Navigation with Spatiotemporal Context Integration

TL;DR

LongFly addresses the challenge of long-horizon UAV vision-and-language navigation by introducing a dedicated history-aware spatiotemporal framework. It combines slot-based historical image compression (SHIC), spatiotemporal trajectory encoding (STE), and prompt-guided multimodal integration (PGM) to fuse visual history, motion priors, and language instructions into a structured prompt for a multimodal LLM. The approach yields consistent improvements over state-of-the-art baselines on the OpenUAV benchmark, including seen and unseen environments, and is supported by ablations that validate the contribution of each module. The results demonstrate that unified spatiotemporal context modeling enhances robustness, semantic grounding, and long-horizon planning in complex 3D environments, with strong generalization to unseen objects and maps.

Abstract

Unmanned aerial vehicles (UAVs) are crucial tools for post-disaster search and rescue, facing challenges such as high information density, rapid changes in viewpoint, and dynamic structures, especially in long-horizon navigation. However, current UAV vision-and-language navigation(VLN) methods struggle to model long-horizon spatiotemporal context in complex environments, resulting in inaccurate semantic alignment and unstable path planning. To this end, we propose LongFly, a spatiotemporal context modeling framework for long-horizon UAV VLN. LongFly proposes a history-aware spatiotemporal modeling strategy that transforms fragmented and redundant historical data into structured, compact, and expressive representations. First, we propose the slot-based historical image compression module, which dynamically distills multi-view historical observations into fixed-length contextual representations. Then, the spatiotemporal trajectory encoding module is introduced to capture the temporal dynamics and spatial structure of UAV trajectories. Finally, to integrate existing spatiotemporal context with current observations, we design the prompt-guided multimodal integration module to support time-based reasoning and robust waypoint prediction. Experimental results demonstrate that LongFly outperforms state-of-the-art UAV VLN baselines by 7.89\% in success rate and 6.33\% in success weighted by path length, consistently across both seen and unseen environments.

Figures (8)

• Figure 1: Effect of spatiotemporal context integration on UAV VLN. Left (red): Navigation based only on the current image and waypoint fails under rapid viewpoint and layout changes. Right (green): LongFly introduces a spatiotemporal context modeling framework for long-horizon UAV VLN. LongFly dynamically distills multi-view historical observations into a compact and semantically rich representation, encodes UAV trajectory dynamics, and aligns spatiotemporal context with current observations and language instructions for robust navigation in complex 3D environments.
• Figure 2: LongFly is a spatiotemporal context modeling framework for long-horizon UAV VLN. It addresses long-horizon reasoning by jointly integrating language instructions, historical visual observations, and flight trajectories. Given a natural language instruction, the framework maps multi-view historical images and past trajectory points into compact representations. Historical visual observations are compressed to extract instruction-relevant semantic cues, while historical trajectories are encoded as explicit motion priors that capture long-horizon path evolution. These visual and trajectory contexts are then fused under the guidance of the instruction and fed into a multimodal model for cross-modal reasoning, enabling consistent waypoint prediction over long distances.
• Figure 3: Overview of the Slot-Based Historical Image Compression (SHIC) module. Multi-view visual observations collected over time are encoded and dynamically aggregated into a fixed number of semantic slots. Through recurrent slot assignment and weighted aggregation, SHIC compresses long-horizon visual histories into compact representations that retain spatial and semantic consistency.
• Figure 4: Structure of the Spatio-Temporal Trajectory Encoding (STE) module. Historical waypoints are encoded into trajectory tokens in temporal order. By combining concatenation projection, time–pose embedding, and a residual MLP, STE models motion continuity and provides spatiotemporal trajectory features for long-horizon navigation.
• Figure 5: Illustration of the prompt-guided multimodal integration in LongFly. The structured prompt consists of three parts: (i) the task instruction, (ii) a Qwen-compatible conversation template, and (iii) UAV history status information, including previous displacement, current position, historical waypoints, historical visual observations, and the current image.