Perception-Enhanced Multitask Multimodal Semantic Communication for UAV-Assisted Integrated Sensing and Communication System

TL;DR

This work tackles efficient multitask multit modal semantic communication for UAV-assisted ISAC under limited bandwidth. It introduces the PE-MMSC framework, where an on-board Perception Enhanced (PE) module guides attention-based fusion of hyperspectral and LiDAR semantics to improve target classification and data reconstruction, even in low-SNR channels. A semant ic codec stack—comprising an encoder, fusion encoder, classifier, and decoder—is trained end-to-end with a joint loss that balances classification accuracy and reconstruction quality, constrained by $NMSE$ definitions such as $NMSE(\boldsymbol{D}_N^{\text{mod}},\hat{\boldsymbol{D}}_N^{\text{mod}})=\frac{\mathbb{E}[\|\boldsymbol{D}_N^{\text{mod}}-\hat{\boldsymbol{D}}_N^{\text{mod}}\|^2]}{\mathbb{E}[\|\boldsymbol{D}_N^{\text{mod}}\|^2]}$ and accuracy $\mathcal{A}(\boldsymbol{C}_N^{\text{fin}},\boldsymbol{C}_N^{\text{true}})$, optimized with Adam. Experiments on the Houston2013 dataset show that PE-MMSC improves target classification accuracy by about 5–10% and reduces reconstruction NMSE (e.g., up to 36% for LiDAR) with a modest computational overhead, demonstrating practical benefits for UAV ISAC deployments.

Abstract

Recent advances in integrated sensing and communication (ISAC) unmanned aerial vehicles (UAVs) have enabled their widespread deployment in critical applications such as emergency management. This paper investigates the challenge of efficient multitask multimodal data communication in UAV-assisted ISAC systems, in the considered system model, hyperspectral (HSI) and LiDAR data are collected by UAV-mounted sensors for both target classification and data reconstruction at the terrestrial BS. The limited channel capacity and complex environmental conditions pose significant challenges to effective air-to-ground communication. To tackle this issue, we propose a perception-enhanced multitask multimodal semantic communication (PE-MMSC) system that strategically leverages the onboard computational and sensing capabilities of UAVs. In particular, we first propose a robust multimodal feature fusion method that adaptively combines HSI and LiDAR semantics while considering channel noise and task requirements. Then the method introduces a perception-enhanced (PE) module incorporating attention mechanisms to perform coarse classification on UAV side, thereby optimizing the attention-based multimodal fusion and transmission. Experimental results demonstrate that the proposed PE-MMSC system achieves 5\%--10\% higher target classification accuracy compared to conventional systems without PE module, while maintaining comparable data reconstruction quality with acceptable computational overheads.

Figures (7)

• Figure 1: Perception-enhanced multitask multimodal semantic communication for UAV-assisted ISAC system
• Figure 2: Attention mechanism of perception enhanced fusion
• Figure 3: Accuracy vs SNR (multi-modal)
• Figure 4: Accuracy vs SNR (single-modal)
• Figure 5: NMSE vs SNR (HSI)