Enhancing Neural Adaptive Wireless Video Streaming via Lower-Layer Information Exposure and Online Tuning

TL;DR

The paper tackles QoE degradation in wireless video streaming by incorporating lower-layer information into a cross-layer DRL framework. It introduces an enhanced A3C (eA3C) that jointly trains policy and value networks offline, leveraging cross-layer and historical data, and then applies two continual-learning online tuning schemes (eA3C-OTP and eA3C-OTPV) to personalize policies for individual users in real time. Empirical results show offline QoE gains of about 6.8%–14.4% over state-of-the-art offline methods, with online tuning further improving QoE by 6%–28% over the offline policy, demonstrating the practical benefits of cross-layer modeling and online adaptation. The approach provides a flexible framework that balances QoE, training time, and inference latency, with significant implications for operator-controlled adaptive video streaming in heterogeneous networks.

Abstract

Deep reinforcement learning (DRL) demonstrates its promising potential in the realm of adaptive video streaming and has recently received increasing attention. However, existing DRL-based methods for adaptive video streaming use only application (APP) layer information, adopt heuristic training methods, and train generalized neural networks with pre-collected data. This paper aims to boost the quality of experience (QoE) of adaptive wireless video streaming by using lower-layer information, deriving a rigorous training method, and adopting online tuning with real-time data. First, we formulate a more comprehensive and accurate adaptive wireless video streaming problem as an infinite stage discounted Markov decision process (MDP) problem by additionally incorporating past and lower-layer information, allowing a flexible tradeoff between QoE and costs for obtaining system information and solving the problem. In the offline scenario (only with pre-collected data), we propose an enhanced asynchronous advantage actor-critic (eA3C) method by jointly optimizing the parameters of parameterized policy and value function. Specifically, we build an eA3C network consisting of a policy network and a value network that can utilize cross-layer, past, and current information and jointly train the eA3C network using pre-collected samples. In the online scenario (with additional real-time data), we propose two continual learning-based online tuning methods for designing better policies for a specific user with different QoE and training time tradeoffs. Finally, experimental results show that the proposed offline policy can improve the QoE by 6.8~14.4% compared to the state-of-arts in the offline scenario, and the proposed online policies can further achieve 6~28% gains in QoE over the proposed offline policy in the online scenario.

Figures (10)

• Figure 1: Relationship between APP layer throughput sequence $\{C_{n}:n=1,\ldots,\}$ and MAC rate sequence $\{X_{n} : n = 1,\ldots,\}$ on two collected datasetsreport (see Section \ref{['sec:sim']} for details). The (sample) cross-correlation between $X$ and $C$ and auto-correlation of $C$ are defined as $R_{X,C}(\tau) \triangleq\frac{1}{N-\tau}\sum^{N-\tau}_{n=1}X_{n}C_{n+\tau}$, $R_{C,C}(\tau) \triangleq\frac{1}{N-\tau}\sum^{N-\tau}_{n=1}C_{n}C_{n+\tau}$, respectively, where $\tau \in \{0,1,\ldots,N\}$ is time lag, and $N$ is the number of samples.
• Figure 2: System model. $D = 3, \mathcal{D} = \{1,2,3\}, \mathcal{R} = \{r_{1},r_{2},r_{3}\}$.
• Figure 3: Structure of the eA3C network
• Figure 4: Structures of the eA3C-OTP and eA3C-OTPV networks.
• Figure 5: APP layer throughput and lower-layer information.

Theorems & Definitions (5)

• Lemma 1: Transition Probabilities
• Proof 1
• Lemma 2: Optimal Value Function and Optimal Stationary Randomized Policy
• Lemma 3: Relationship between the Problems in \ref{['prob:offline_policy_gradient_formulation']} and \ref{['prob:policy_gradient_equal']}
• Proof 2