Offline to Online Learning for Real-Time Bandwidth Estimation

TL;DR

Real-time video requires accurate bandwidth estimation, but heterogeneous networks limit the effectiveness of traditional heuristics and online RL can be data-inefficient. Merlin transforms a heuristic bandwidth estimator (UKF) into a neural policy via Behavioral Cloning on offline telemetry and then personalizes it online with PPO using a QoE-based reward, enabling data-efficient end-user customization. It generalizes from offline demonstrations to real intercontinental videoconferencing, matching UKF QoE while delivering up to 7.8% QoE gains through online finetuning and requiring ≈80% fewer samples than online RL. This approach combines domain-knowledge with data-driven learning to provide practical, scalable, and personalized BWE for real-time video applications.

Abstract

Real-time video applications require accurate bandwidth estimation (BWE) to maintain user experience across varying network conditions. However, increasing network heterogeneity challenges general-purpose BWE algorithms, necessitating solutions that adapt to end-user environments. While widely adopted, heuristic-based methods are difficult to individualize without extensive domain expertise. Conversely, online reinforcement learning (RL) offers ease of customization but neglects prior domain expertise and suffers from sample inefficiency. Thus, we present Merlin, an imitation learning-based solution that replaces the manual parameter tuning of heuristic-based methods with data-driven updates to streamline end-user personalization. Our key insight is that transforming heuristic-based BWE algorithms into neural networks facilitates data-driven personalization. Merlin utilizes Behavioral Cloning to efficiently learn from offline telemetry logs, capturing heuristic policies without live network interactions. The cloned policy can then be seamlessly tailored to end user network conditions through online finetuning. In real intercontinental videoconferencing calls, Merlin matches our heuristic's policy with no statistically significant differences in user quality of experience (QoE). Finetuning Merlin's control policy to end-user environments enables QoE improvements of up to 7.8% compared to the heuristic policy. Lastly, our IL-based design performs competitively with current state-of-the-art online RL techniques but converges with 80% fewer videoconferencing samples, facilitating practical end-user personalization.

Figures (8)

• Figure 1: Learning from Offline Demonstrations.
• Figure 2: Media Stack Overview.
• Figure 3: Merlin's Network Architecture.
• Figure 4: Performance Comparison of Merlin, UKF, and Hybrid Methods during Videoconferencing Calls.
• Figure 5: Personalizing the Heuristic Policy to LBW and HBW environments.