Distributed Radiance Fields for Edge Video Compression and Metaverse Integration in Autonomous Driving

TL;DR

This paper addresses the challenge of real-time metaverse updates for autonomous driving by reducing data transmission over MEC networks. It introduces a distributed Radiance Field (RF) framework that uses RF encoders/decoders to render scene views and conveys only a delta via H.264 (no I-frames), leveraging synchronized RFs at sender and receiver to maintain visual fidelity. Two RF backbones are explored—NeRF-based representations and 3D Gaussian Splatting (3DGS)—with 3DGS delivering superior reconstruction quality in many scenarios. Experimental evaluation on CARLA-derived urban scenes shows significant data savings (up to 80%) and high perceptual quality (PSNR/SSIM, LPIPS) for RF-based video compression, highlighting the approach’s potential for scalable, low-latency metaverse or digital twin integration in edge-powered autonomous mobility.

Abstract

The metaverse is a virtual space that combines physical and digital elements, creating immersive and connected digital worlds. For autonomous mobility, it enables new possibilities with edge computing and digital twins (DTs) that offer virtual prototyping, prediction, and more. DTs can be created with 3D scene reconstruction methods that capture the real world's geometry, appearance, and dynamics. However, sending data for real-time DT updates in the metaverse, such as camera images and videos from connected autonomous vehicles (CAVs) to edge servers, can increase network congestion, costs, and latency, affecting metaverse services. Herein, a new method is proposed based on distributed radiance fields (RFs), multi-access edge computing (MEC) network for video compression and metaverse DT updates. RF-based encoder and decoder are used to create and restore representations of camera images. The method is evaluated on a dataset of camera images from the CARLA simulator. Data savings of up to 80% were achieved for H.264 I-frame - P-frame pairs by using RFs instead of I-frames, while maintaining high peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) qualitative metrics for the reconstructed images. Possible uses and challenges for the metaverse and autonomous mobility are also discussed.

Figures (6)

• Figure 1: Proposed novel video compression using distributed RFs in MEC network. The diagram shows a) the real-world scene with CAVs and MEC infrastructure b) RF encoder preparing the delta with differences between the real and RF frame c) RF decoder reapplying the delta to RF frame rendered by local copy of the RF.
• Figure 2: GT images from CARLA simulator with matched INGP and 3DGS images from the same camera pose. Note the varying degree of blur and missing details in INGP and 3DGS rendered frames, like distortion of the letters in the "MUSEUM" sign and missing parts of the lamp structures.
• Figure 3: Encoder setting-averaged compression savings relative to H.264 achieved for individual images in empty scene scenario using INGP.
• Figure 4: Encoder setting-averaged compression savings relative to H.264 achieved for individual images in empty scene scenario using 3DGS.
• Figure 5: Encoder setting-averaged compression savings relative to H.264 achieved for individual images in vehicles in the scene scenario using INGP.