Human-in-the-Loop Bandwidth Estimation for Quality of Experience Optimization in Real-Time Video Communication

TL;DR

This work addresses the challenge of delivering high QoE in real-time video communication by learning a bandwidth estimator aligned with user-perceived quality. It combines human-in-the-loop QoE reward models trained from ITU-T guidelines with a distributional offline RL framework (DIQL) that uses a multi-modal, asymmetric actor-critic over a POMDP formulation to infer bandwidth from partial observations. Trained on roughly 1 million real-world Microsoft Teams traces, the approach is deployed in production and evaluated via a staged A/B rollout, achieving an $11.41\%$ reduction in subjective poor-call rate and modest QoE gains, while demonstrating robust generalization on D4RL benchmarks. The results validate the viability of offline, QoE-driven policy learning for latency-sensitive systems and suggest broader applicability to other real-time networked multimedia applications.

Abstract

The quality of experience (QoE) delivered by video conferencing systems is significantly influenced by accurately estimating the time-varying available bandwidth between the sender and receiver. Bandwidth estimation for real-time communications remains an open challenge due to rapidly evolving network architectures, increasingly complex protocol stacks, and the difficulty of defining QoE metrics that reliably improve user experience. In this work, we propose a deployed, human-in-the-loop, data-driven framework for bandwidth estimation to address these challenges. Our approach begins with training objective QoE reward models derived from subjective user evaluations to measure audio and video quality in real-time video conferencing systems. Subsequently, we collect roughly $1$M network traces with objective QoE rewards from real-world Microsoft Teams calls to curate a bandwidth estimation training dataset. We then introduce a novel distributional offline reinforcement learning (RL) algorithm to train a neural-network-based bandwidth estimator aimed at improving QoE for users. Our real-world A/B test demonstrates that the proposed approach reduces the subjective poor call ratio by $11.41\%$ compared to the baseline bandwidth estimator. Furthermore, the proposed offline RL algorithm is benchmarked on D4RL tasks to demonstrate its generalization beyond bandwidth estimation.

Figures (2)

• Figure 1: Bandwidth estimation in RTC: two endpoints exchange media over a time-varying bottleneck link. Cross-traffic reduces the available bandwidth. The reciever client infers available bandwidth from packet statistics (rate, delay, jitter, loss), which is fed back to the sender client to set encoder target bitrates. Accurate estimation is critical for optimising QoE, as it directly impacts video smoothness, audio clarity, and overall user satisfaction.
• Figure 2: Overall framework: (1) QoE reward modeling from human feedback (P.808/P.910) produces in-stack audio/video quality predictors for use as a QoE reward; (2) real-call telemetry (RTP headers, baseline bandwidth estimates) is transformed into $(o_t,a_t,r_t,o_{t+1})$ trajectories; (3) a distributional offline RL agent (value $Z_V$, critic $Z_Q$, policy $\pi$) is trained and exported to ONNX for client-side inference; (4) telemetry and A/B testing close the deployment loop.