Goal-oriented Semantic Communications for Metaverse Construction via Generative AI and Optimal Transport
Zhe Wang, Nan Li, Yansha Deng, A. Hamid Aghvami
TL;DR
This work addresses the bandwidth and latency challenges of real-time metaverse construction by proposing a goal-oriented semantic communication framework that extracts and transmits semantic content via a HgNet semantic encoder and an OT-enabled denoiser, then reconstructs the scenery using Stable Diffusion and NeRF. It introduces a structured knowledge base at the receiver and a relaxed OT optimization with semantic selective correction to align semantic distributions under wireless fading. The approach yields substantial gains in key-point accuracy (KPE), 3D viewing consistency (P2Point), and transmission latency, outperforming traditional image-based transmission, especially in adverse channel conditions. The proposed framework enables more reliable, low-latency metaverse construction suitable for industrial and interactive VR/AR scenarios, with practical impact on next-generation semantic-aware wireless systems.
Abstract
The emergence of the metaverse has boosted productivity and creativity, driving real-time updates and personalized content, which will substantially increase data traffic. However, current bit-oriented communication networks struggle to manage this high volume of dynamic information, restricting metaverse applications interactivity. To address this research gap, we propose a goal-oriented semantic communication (GSC) framework for metaverse. Building on an existing metaverse wireless construction task, our proposed GSC framework includes an hourglass network-based (HgNet) encoder to extract semantic information of objects in the metaverse; and a semantic decoder that uses this extracted information to reconstruct the metaverse content after wireless transmission, enabling efficient communication and real-time object behaviour updates to the scenery for metaverse construction task. To overcome the wireless channel noise at the receiver, we design an optimal transport (OT)-enabled semantic denoiser, which enhances the accuracy of metaverse scenery through wireless communication. Experimental results show that compared to the conventional metaverse construction, our proposed GSC framework significantly reduces wireless metaverse construction latency by 92.6\%, while improving metaverse object status accuracy and viewing experience by 45.6\% and 44.7\%, respectively.