Neural-Enhanced Rate Adaptation and Computation Distribution for Emerging mmWave Multi-User 3D Video Streaming Systems

TL;DR

This paper addresses efficient delivery of high-quality 360° video for multi-user VR over mmWave by jointly optimizing communication and edge computing resources. It introduces a deep reinforcement learning framework (MTRC) for multitask rate adaptation and computation distribution, augmented by neural network cascades (R1C2 and C1R2) to capture interdependencies between decisions. Using a trace-driven, multi-user VR simulator with real mmWave traces and UHD 360° video data, the authors demonstrate that C1R2 yields the strongest QoE improvements, including 5.21–6.06 dB PSNR gains and substantial reductions in rebuffering time and quality variation. The results indicate the approach can significantly improve end-user experience in edge-assisted VR streaming by balancing throughput, decoding/rendering workload, and viewport requirements.

Abstract

We investigate multitask edge-user communication-computation resource allocation for $360^\circ$ video streaming in an edge-computing enabled millimeter wave (mmWave) multi-user virtual reality system. To balance the communication-computation trade-offs that arise herein, we formulate a video quality maximization problem that integrates interdependent multitask/multi-user action spaces and rebuffering time/quality variation constraints. We formulate a deep reinforcement learning framework for \underline{m}ulti-\underline{t}ask \underline{r}ate adaptation and \underline{c}omputation distribution (MTRC) to solve the problem of interest. Our solution does not rely on a priori knowledge about the environment and uses only prior video streaming statistics (e.g., throughput, decoding time, and transmission delay), and content information, to adjust the assigned video bitrates and computation distribution, as it observes the induced streaming performance online. Moreover, to capture the task interdependence in the environment, we leverage neural network cascades to extend our MTRC method to two novel variants denoted as R1C2 and C1R2. We train all three methods with real-world mmWave network traces and $360^\circ$ video datasets to evaluate their performance in terms of expected quality of experience (QoE), viewport peak signal-to-noise ratio (PSNR), rebuffering time, and quality variation. We outperform state-of-the-art rate adaptation algorithms, with C1R2 showing best results and achieving $5.21-6.06$ dB PSNR gains, $2.18-2.70$x rebuffering time reduction, and $4.14-4.50$ dB quality variation reduction.

Figures (7)

• Figure 1: Edge-assisted VR system model: Multiple VR headsets connected to an edge computing unit (ECU) via a mmWave network.
• Figure 2: Multi-layer $360^\circ$ video model (lower left corner): Viewport-specific enhancement layers combined with a wide $360^\circ$ panorama baseline layer are transmitted to a VR headset via a mmWave wireless link. As the number of added enhancement layers increases, the video bitrate and the video quality delivered to the VR user increase.
• Figure 3: VR headset playback buffer dynamics.
• Figure 4: Our MTRC architecture comprises actor and critic networks. Based on the observed state information for each user, the actor network makes a joint rate adaptation and computation distribution decision, and the critic network estimates the state values.
• Figure 5: (a) MTRC makes a joint decision on rate adaptation and computation distribution action. (b) and (c) R1C2 and C1R2 employ neural network cascades to capture the interdependence between the rate adaptation and computation distribution actions.