RAPTOR: Real-Time High-Resolution UAV Video Prediction with Efficient Video Attention

TL;DR

RAPTOR tackles the real-time, high-resolution video prediction bottleneck for UAVs by introducing Efficient Video Attention (EVA), a spatiotemporal translator that factorizes along spatial and temporal axes to achieve linear-time and linear-memory complexity. Coupled with an Encoder--Translator--Decoder architecture and a three-stage curriculum, RAPTOR delivers high-fidelity predictions at 512^2 and 1024^2 resolutions in real time, enabling safer, anticipatory autonomous navigation on edge hardware. The approach sets new state-of-the-art results on UAVid and KTH benchmarks, and demonstrates a substantial 18% improvement in real-world UAV navigation success. This work advances practical embodied perception by enabling scalable, high-resolution forecasting that informs planning under tight latency constraints.

Abstract

Video prediction is plagued by a fundamental trilemma: achieving high-resolution and perceptual quality typically comes at the cost of real-time speed, hindering its use in latency-critical applications. This challenge is most acute for autonomous UAVs in dense urban environments, where foreseeing events from high-resolution imagery is non-negotiable for safety. Existing methods, reliant on iterative generation (diffusion, autoregressive models) or quadratic-complexity attention, fail to meet these stringent demands on edge hardware. To break this long-standing trade-off, we introduce RAPTOR, a video prediction architecture that achieves real-time, high-resolution performance. RAPTOR's single-pass design avoids the error accumulation and latency of iterative approaches. Its core innovation is Efficient Video Attention (EVA), a novel translator module that factorizes spatiotemporal modeling. Instead of processing flattened spacetime tokens with $O((ST)^2)$ or $O(ST)$ complexity, EVA alternates operations along the spatial (S) and temporal (T) axes. This factorization reduces the time complexity to $O(S + T)$ and memory complexity to $O(max(S, T))$, enabling global context modeling at $512^2$ resolution and beyond, operating directly on dense feature maps with a patch-free design. Complementing this architecture is a 3-stage training curriculum that progressively refines predictions from coarse structure to sharp, temporally coherent details. Experiments show RAPTOR is the first predictor to exceed 30 FPS on a Jetson AGX Orin for $512^2$ video, setting a new state-of-the-art on UAVid, KTH, and a custom high-resolution dataset in PSNR, SSIM, and LPIPS. Critically, RAPTOR boosts the mission success rate in a real-world UAV navigation task by 18/%, paving the way for safer and more anticipatory embodied agents.

Figures (4)

• Figure 1: RAPTOR enables scalable, high-fidelity video prediction for demanding, high-resolution scenarios. (a, b) Benchmarked on an NVIDIA RTX 6000 Ada (48GB), RAPTOR's linear complexity design maintains a low memory footprint, making it the first framework to efficiently scale beyond $1024^2$ resolution as competitors hit memory ceilings (Out-of-Memory). (c, d) In contrast, conventional methods relying on pixel-wise losses often result in (c) temporal tearing and (d) edge blurring that obscures vital targets. (e) The proposed RAPTOR avoids these pitfalls, generating sharp and coherent predictions that preserve crucial details for reliable downstream planning.
• Figure 2: Overview of RAPTOR architecture. The framework follows an Encoder--Translator--Decoder design. The EVA Translator employs a patch-free design, operating on full feature maps from the encoder and stacking $N$ EVA blocks. Each block alternates a temporal TimeMix and a spatial SpaceMix built from specialized, asymmetric operators, achieving global spatiotemporal receptive fields with linear-in-length complexity and enabling real-time high-resolution prediction.
• Figure 3: Qualitative comparison on a challenging UAVid scene. The magnified insets highlight how RAPTOR, particularly with the full S1+S2+S3 curriculum, generates sharper and more coherent predictions of moving vehicles compared to baselines. RAPTOR is also the only model capable of producing a prediction at $1024^2$ resolution.
• Figure 4: Qualitative comparison from the real-world UAV navigation task. Magnified insets highlight that RAPTOR (with full Three-Stage Curriculum) outperforms the baselines by generating sharp and coherent predictions of the moving pedestrian, even in a nighttime scenario.