On the Use of Immersive Digital Technologies for Designing and Operating UAVs

TL;DR

The paper surveys immersive digital technologies for UAVs, detailing how Digital Twin (DT) and Extended Reality (XR) can bridge the simulation-to-reality gap and enhance safety, collaboration, and operator awareness. It analyzes how DT integrates with machine learning, notably deep reinforcement learning (DRL), to improve UAV design, control, and swarm behavior, while XR enhances perception, navigation, and training; Generative AI (GAI) is identified as a key enabler for data augmentation, content generation, and adaptive interactions. Practical challenges such as computational overhead, data synchronization, cybersecurity, privacy, and ethical considerations are examined, with proposed countermeasures including GAI-driven data synthesis, Federated Learning, and explainable AI (XAI). The authors outline future directions, emphasizing explainable AI, VR interfaces and training, closer fusion of ML with DTs, and deeper integration of GAI with XR and DTs to advance safe, scalable, and intelligent UAV operations across single and swarming platforms.

Abstract

Unmanned Aerial Vehicles (UAVs) offer agile, secure and efficient solutions for communication relay networks. However, their modeling and control are challenging, and the mismatch between simulations and actual conditions limits real-world deployment. Moreover, improving situational awareness is essential. Several studies proposed integrating the operation of UAVs with immersive digital technologies such as Digital Twin (DT) and Extended Reality (XR) to overcome these challenges. This paper provides a comprehensive overview of the latest research and developments involving immersive digital technologies for UAVs. We explore the use of Machine Learning (ML) techniques, particularly Deep Reinforcement Learning (DRL), to improve the capabilities of DT for UAV systems. We provide discussion, identify key research gaps, and propose countermeasures based on Generative AI (GAI), emphasizing the significant role of AI in advancing DT technology for UAVs. Furthermore, we review the literature, provide discussion, and examine how the XR technology can transform UAV operations with the support of GAI, and explore its practical challenges. Finally, we propose future research directions to further develop the application of immersive digital technologies for UAV operation.

Figures (2)

• Figure 1: An overview of UAV swarm management system integrating AI, DT, and AR technologies. The DT optimizes and controls swarm behavior using artificial intelligence. AR overlays essential flight information for real-time monitoring and planning.
• Figure 2: Immersive digital technologies for UAVs, where DT integrated with ML contributes to UAV safety, traffic optimization, QoS, design, reduced simulation-reality gap, and intelligent cooperation. Also, XR with ML contributes to situational awareness, safety, mission success rate, navigation and collision avoidance, efficient decision-making, and training operators.