Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Immersive Media and Massive Twinning: Advancing Towards the Metaverse

Wassim Hamidouche, Lina Bariah, Merouane Debbah

TL;DR

The paper addresses the challenge of delivering fully immersive Metaverse experiences via massive twinning by systematically reviewing immersive media modalities, their 3D representations, and the coding/transport standards that enable real-time streaming. It argues that 6G will support inter- and intra-twin communications through semantic and goal-oriented paradigms, complemented by holographic-type communication to realize ultra-low latency, high-fidelity holographic streams. The authors synthesize existing standards (e.g., MIV, V-PCC, G-PCC, v-dmc, OMAF) and emerging AI approaches (NeRF, generative AI, LLMs) while identifying key open challenges in standardization, AI integration, edge computing, tactile Internet, ethics, and QoE, providing a roadmap for future work. The work highlights that achieving Metaverse-worth QoE requires co-design of media modalities, coding, transport, AI-driven protocols, and 6G-enabled network architectures, with potential latency below $<1\mathrm{ms}$ and multi-Gbps throughput in practical deployments.

Abstract

The advent of the Metaverse concept has further expedited the evolution of haptic, tactile internet, and multimedia applications with their VR/AR/XR services, and therefore, fully-immersive sensing is most likely to define the next generation of wireless networks as a key to realize the speculative vision of the Metaverse. In this magazine, we articulate different types of media that we envision will be communicated between the cyber and physical twins in the Metaverse. In particular, we explore the advantages grasped by exploiting each kind, and we point out critical challenges pertinent to 3D data processing, coding, transporting, and rendering. We further shed light on the role of future wireless networks in delivering the anticipated quality of immersion through the reliable streaming of multimedia signals between the digital twin and its physical counterpart. Specifically, we explore emergent communication paradigms, including semantic, holographic, and goal-oriented communication, which we expect to realize energy and spectrally efficient Metaverse while ensuring ultra-low latency.

Immersive Media and Massive Twinning: Advancing Towards the Metaverse

TL;DR

The paper addresses the challenge of delivering fully immersive Metaverse experiences via massive twinning by systematically reviewing immersive media modalities, their 3D representations, and the coding/transport standards that enable real-time streaming. It argues that 6G will support inter- and intra-twin communications through semantic and goal-oriented paradigms, complemented by holographic-type communication to realize ultra-low latency, high-fidelity holographic streams. The authors synthesize existing standards (e.g., MIV, V-PCC, G-PCC, v-dmc, OMAF) and emerging AI approaches (NeRF, generative AI, LLMs) while identifying key open challenges in standardization, AI integration, edge computing, tactile Internet, ethics, and QoE, providing a roadmap for future work. The work highlights that achieving Metaverse-worth QoE requires co-design of media modalities, coding, transport, AI-driven protocols, and 6G-enabled network architectures, with potential latency below and multi-Gbps throughput in practical deployments.

Abstract

The advent of the Metaverse concept has further expedited the evolution of haptic, tactile internet, and multimedia applications with their VR/AR/XR services, and therefore, fully-immersive sensing is most likely to define the next generation of wireless networks as a key to realize the speculative vision of the Metaverse. In this magazine, we articulate different types of media that we envision will be communicated between the cyber and physical twins in the Metaverse. In particular, we explore the advantages grasped by exploiting each kind, and we point out critical challenges pertinent to 3D data processing, coding, transporting, and rendering. We further shed light on the role of future wireless networks in delivering the anticipated quality of immersion through the reliable streaming of multimedia signals between the digital twin and its physical counterpart. Specifically, we explore emergent communication paradigms, including semantic, holographic, and goal-oriented communication, which we expect to realize energy and spectrally efficient Metaverse while ensuring ultra-low latency.
Paper Structure (20 sections, 6 figures, 4 tables)

This paper contains 20 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Immersive Media Streaming
  3. 3D visual signal
  4. Visual 3D visual signal coding
  5. 3D video streaming
  6. 6G: Emerging Technologies for Immersive Media Communication
  7. Beyond Shannon Communication
  8. Goal-Oriented Communication
  9. Semantic Communications
  10. The Interplay of Goal- & Semantic-Aware Communication
  11. Communication Protocols
  12. Holographic-Type Communication
  13. Open Challenges & Future Research Directions
  14. Standardization
  15. AI for the Metaverse
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Immersive Media for Digital Twin: Opportunities, Limitations, 6G Technologies.
  • Figure 2: Viewing degrees of freedom (DoF) enabled by 3D visual modalities.
  • Figure 3: Implicit-based coding of 2D views for continuous 3D representation of the scene.
  • Figure 4: ITU-T and ISO/IEC standards for compression and carriage of 3D visual modalities. Dashed correspond to ongoing standardization activities.
  • Figure 5: Architecture of goal-oriented, semantic, and goal-oriented semantic communication.
  • ...and 1 more figures