Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTC

TL;DR

This work proposes a novel post-deployment shaping of policies (Streetwise), conditioned on real-time characterization of out-of-distribution sub-spaces that leads to robust actions in bandwidth estimation (BWE) of network bottlenecks in RTC and in standard benchmarks.

Abstract

The difficulty of exploring and training online on real production systems limits the scope of real-time online data/feedback-driven decision making. The most feasible approach is to adopt offline reinforcement learning from limited trajectory samples. However, after deployment, such policies fail due to exogenous factors that temporarily or permanently disturb/alter the transition distribution of the assumed decision process structure induced by offline samples. This results in critical policy failures and generalization errors in sensitive domains like Real-Time Communication (RTC). We solve this crucial problem of identifying robust actions in presence of domain shifts due to unseen exogenous stochastic factors in the wild. As it is impossible to learn generalized offline policies within the support of offline data that are robust to these unseen exogenous disturbances, we propose a novel post-deployment shaping of policies (Streetwise), conditioned on real-time characterization of out-of-distribution sub-spaces. This leads to robust actions in bandwidth estimation (BWE) of network bottlenecks in RTC and in standard benchmarks. Our extensive experimental results on BWE and other standard offline RL benchmark environments demonstrate a significant improvement ($\approx$ 18% on some scenarios) in final returns wrt. end-user metrics over state-of-the-art baselines.

Figures (3)

• Figure 1: Example calls with exogenous disturbances leading to sub-optimal predictions by deployed policies in Bandwidth Estimation domain
• Figure 2: Streetwise agent architecture. Left shows the learning/training phase on limited offline samples, where Offline RL algorithm trains a policy $\pi_\mathcal{D}$ and also preserves the value function $Q_\mathcal{D}$. We also train an Autoencoder that will detect and quantify OOD after deployment. Right shows the post-deployment framework, where new given new observations from both offline and OOD transition distributions and state distributions employs the OOD/Disturbance detector to compute reconstruction loss as metric for the position, span and amplitude of OOD regions. It then shapes/perturbs the predicted action with a shaping potential $\phi$ based on the gradient of the Value function and the reconstruction loss
• Figure 3: Comparison of model predicted bandwidths and video MOS on calls with the Random Loss network profile.