Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotion

TL;DR

Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotion presents a CVaR-constrained RL framework to train multiple risk-aware policies across risk levels $\alpha\in(0,1)$ and employs an online Empirical-Bernstein UCB bandit to select the best policy based on observed episodic returns. The method combines a clipped-Lagrangian PPO objective with bootstrapped returns to stabilize training, and uses bandit-based online adaptation to match deployment conditions without privileged information. In both simulation and hardware, the approach nearly doubles mean and tail performance in unseen environments and the bandit rapidly identifies the best risk level within about two minutes of operation. This yields robust, adaptive quadruped locomotion under unknown disturbances, enabling effective deployment without extensive manual tuning of risk settings.

Abstract

In this work, we study risk-aware reinforcement learning for quadrupedal locomotion. Our approach trains a family of risk-conditioned policies using a Conditional Value-at-Risk (CVaR) constrained policy optimization technique that provides improved stability and sample efficiency. At deployment, we adaptively select the best performing policy from the family of policies using a multi-armed bandit framework that uses only observed episodic returns, without any privileged environment information, and adapts to unknown conditions on the fly. Hence, we train quadrupedal locomotion policies at various levels of robustness using CVaR and adaptively select the desired level of robustness online to ensure performance in unknown environments. We evaluate our method in simulation across eight unseen settings (by changing dynamics, contacts, sensing noise, and terrain) and on a Unitree Go2 robot in previously unseen terrains. Our risk-aware policy attains nearly twice the mean and tail performance in unseen environments compared to other baselines and our bandit-based adaptation selects the best-performing risk-aware policy in unknown terrain within two minutes of operation.

Figures (6)

• Figure 1: Overview: We train multiple policies using CVaR-constrained policy optimization, where each critic focuses on a different tail of the return distribution, resulting in policies with varying levels of risk awareness. An upper confidence bound (UCB) bandit is then used to adaptively select the appropriate risk level online. The bandit selects the best performing policy over time as the robot interacts with the environment repeatedly.
• Figure 2: Training performance at $\alpha=0.25$. Curves show mean undiscounted return versus timesteps of environment interactions; solid lines are the mean over 10 runs trained for $30{,}000$ timesteps, shaded regions denote $\pm1$ standard deviation. Evaluations are in the training environment.
• Figure 3: Performance of with different $\alpha$-level. We trained policies different $\alpha$-level and test their performances in perturbed environments. The descriptions of the environments are available at Table \ref{['tab:env']}. For each $\alpha$-level we trained 10 policies using different seeds, and we evaluate each policy with 1000 parallel simulations and report the mean return over 6000 timesteps. We reset the environment and randomize the state of robots every 100 timesteps
• Figure 4: We commanded the robot to walk at full speed ($v=1\,\text{m/s}$) on both ramp and grass terrains. The $\alpha=0.05$ policy did not advance despite forward commands, either when climbing the ramp or when a foot became trapped in a soccer cone hidden beneath the grass. In contrast, the $\alpha=0.25$ policy adapted a stable gait with larger steps, enabling the robot to step over hidden cones. The PPO baseline moved faster but was more prone to losing balance, particularly when descending the ramp or when one of its foot hits the soccer cone.
• Figure 5: Cumulative probability of selecting each policy in one ramp experiment. Selection of policy with $\alpha = 0.25$ quickly dominates after 2000 timesteps, which is roughly 1 minute wall clock time.