Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DQ-Ladder: A Deep Reinforcement Learning-based Bitrate Ladder for Adaptive Video Streaming

Reza Farahani, Zoha Azimi, Vignesh V Menon, Hermann Hellwagner, Radu Prodan, Schahram Dustdar, Christian Timmerer

Abstract

Adaptive streaming of segmented video over HTTP typically relies on a predefined set of bitrate-resolution pairs, known as a bitrate ladder. However, fixed ladders often overlook variations in content and decoding complexities, leading to suboptimal trade-offs between encoding time, decoding efficiency, and video quality. This article introduces DQ-Ladder, a deep reinforcement learning (DRL)-based scheme for constructing time- and quality-aware bitrate ladders for adaptive video streaming applications. DQ-Ladder employs predicted decoding time, quality scores, and bitrate levels per segment as inputs to a Deep Q-Network (DQN) agent, guided by a weighted reward function of decoding time, video quality, and resolution smoothness. We leverage machine learning models to predict decoding time, bitrate level, and objective quality metrics (VMAF, XPSNR), eliminating the need for exhaustive encoding or quality metric computation. We evaluate DQ-Ladder using the Versatile Video Coding (VVC) toolchain (VVenC/VVdeC) on 750 video sequences across six Apple HLS-compliant resolutions and 41 quantization parameters. Experimental results against four baselines show that DQ-Ladder achieves BD-rate reductions of at least 10.3% for XPSNR compared to the HLS ladder, while reducing decoding time by 22%. DQ-Ladder shows significantly lower sensitivity to prediction errors than competing methods, remaining robust even with up to 20% noise.

DQ-Ladder: A Deep Reinforcement Learning-based Bitrate Ladder for Adaptive Video Streaming

Abstract

Adaptive streaming of segmented video over HTTP typically relies on a predefined set of bitrate-resolution pairs, known as a bitrate ladder. However, fixed ladders often overlook variations in content and decoding complexities, leading to suboptimal trade-offs between encoding time, decoding efficiency, and video quality. This article introduces DQ-Ladder, a deep reinforcement learning (DRL)-based scheme for constructing time- and quality-aware bitrate ladders for adaptive video streaming applications. DQ-Ladder employs predicted decoding time, quality scores, and bitrate levels per segment as inputs to a Deep Q-Network (DQN) agent, guided by a weighted reward function of decoding time, video quality, and resolution smoothness. We leverage machine learning models to predict decoding time, bitrate level, and objective quality metrics (VMAF, XPSNR), eliminating the need for exhaustive encoding or quality metric computation. We evaluate DQ-Ladder using the Versatile Video Coding (VVC) toolchain (VVenC/VVdeC) on 750 video sequences across six Apple HLS-compliant resolutions and 41 quantization parameters. Experimental results against four baselines show that DQ-Ladder achieves BD-rate reductions of at least 10.3% for XPSNR compared to the HLS ladder, while reducing decoding time by 22%. DQ-Ladder shows significantly lower sensitivity to prediction errors than competing methods, remaining robust even with up to 20% noise.
Paper Structure (45 sections, 6 equations, 11 figures, 9 tables, 2 algorithms)

This paper contains 45 sections, 6 equations, 11 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. DQ-Ladder Model
  4. Environment
  5. State space
  6. Action space
  7. Reward function
  8. Encoding constraints
  9. DQ-Ladder Architecture
  10. DQN agent operation
  11. DQN experience memory
  12. DQN output
  13. DQ-Ladder Execution Workflow
  14. DQN training
  15. DQN inference
  16. ...and 30 more sections

Figures (11)

  • Figure 1: DQ-Ladder system architecture.
  • Figure 2: Content complexity feature distribution in the dataset.
  • Figure 3: SHAP analysis of feature contribution and interaction across prediction metrics.
  • Figure 4: Distribution of normalized decoding time and quality metrics (XPSNR, VMAF).
  • Figure 5: Convergence trends of the DQN agent for (a) cumulative reward, (b) XPSNR-based quality reward, (c) decoding time penalty, and (d) resolution smoothness penalty.
  • ...and 6 more figures