Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments

Yun Qu, Qi Cheems Wang, Yixiu Mao, Yiqin Lv, Xiangyang Ji

TL;DR

This work addresses robust adaptation in randomized environments by formulating robust active task sampling (RATS) and introducing PDTS, which combines posterior sampling with diversity regularization to select challenging yet diverse task subsets. A risk-predictive model surrogate guides amortized evaluation, and an i-MAB–theoretic view connects PDTS to existing MPTS approaches while mitigating concentration issues. Through extensive meta-RL and robotics domain-randomization experiments, PDTS demonstrates superior CVaR-based robustness, zero-shot and few-shot adaptation gains, and in some cases faster training, across continuous control, physical, and visual domains. The approach offers a practical, scalable, and plug-and-play enhancement for risk-averse sequential decision-making with broad potential impact in robotics and beyond.

Abstract

Task robust adaptation is a long-standing pursuit in sequential decision-making. Some risk-averse strategies, e.g., the conditional value-at-risk principle, are incorporated in domain randomization or meta reinforcement learning to prioritize difficult tasks in optimization, which demand costly intensive evaluations. The efficiency issue prompts the development of robust active task sampling to train adaptive policies, where risk-predictive models are used to surrogate policy evaluation. This work characterizes the optimization pipeline of robust active task sampling as a Markov decision process, posits theoretical and practical insights, and constitutes robustness concepts in risk-averse scenarios. Importantly, we propose an easy-to-implement method, referred to as Posterior and Diversity Synergized Task Sampling (PDTS), to accommodate fast and robust sequential decision-making. Extensive experiments show that PDTS unlocks the potential of robust active task sampling, significantly improves the zero-shot and few-shot adaptation robustness in challenging tasks, and even accelerates the learning process under certain scenarios. Our project website is at https://thu-rllab.github.io/PDTS_project_page.

Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments

TL;DR

This work addresses robust adaptation in randomized environments by formulating robust active task sampling (RATS) and introducing PDTS, which combines posterior sampling with diversity regularization to select challenging yet diverse task subsets. A risk-predictive model surrogate guides amortized evaluation, and an i-MAB–theoretic view connects PDTS to existing MPTS approaches while mitigating concentration issues. Through extensive meta-RL and robotics domain-randomization experiments, PDTS demonstrates superior CVaR-based robustness, zero-shot and few-shot adaptation gains, and in some cases faster training, across continuous control, physical, and visual domains. The approach offers a practical, scalable, and plug-and-play enhancement for risk-averse sequential decision-making with broad potential impact in robotics and beyond.

Abstract

Task robust adaptation is a long-standing pursuit in sequential decision-making. Some risk-averse strategies, e.g., the conditional value-at-risk principle, are incorporated in domain randomization or meta reinforcement learning to prioritize difficult tasks in optimization, which demand costly intensive evaluations. The efficiency issue prompts the development of robust active task sampling to train adaptive policies, where risk-predictive models are used to surrogate policy evaluation. This work characterizes the optimization pipeline of robust active task sampling as a Markov decision process, posits theoretical and practical insights, and constitutes robustness concepts in risk-averse scenarios. Importantly, we propose an easy-to-implement method, referred to as Posterior and Diversity Synergized Task Sampling (PDTS), to accommodate fast and robust sequential decision-making. Extensive experiments show that PDTS unlocks the potential of robust active task sampling, significantly improves the zero-shot and few-shot adaptation robustness in challenging tasks, and even accelerates the learning process under certain scenarios. Our project website is at https://thu-rllab.github.io/PDTS_project_page.
Paper Structure (50 sections, 4 theorems, 38 equations, 13 figures, 4 tables, 5 algorithms)

This paper contains 50 sections, 4 theorems, 38 equations, 13 figures, 4 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Research Background
  3. Task Robust Optimization Methods
  4. RATS Preliminaries & MPTS Modules
  5. Theoretical Investigations and Practical Enhancements
  6. Enable Robust Active Task Sampling with i-MABs
  7. Acquisition Functions Matter in Improving Coverage & Boosting Robustness
  8. Practical Sampling with Stochastic Optimism
  9. Overall Implementation
  10. Experiments
  11. Meta Reinforcement Learning
  12. Physical Robotics Domain Randomization
  13. Visual Robotics Domain Randomization
  14. Conclusion
  15. Adaptive Decision-Making in Randomized Environments
  16. ...and 35 more sections

Key Result

Proposition 3.2

Executing MPTS pipeline in Eq. (eq_mpts_workflows) is equivalent to approximately solving $\mathcal{M}$ with the i-MAB under the UCB principle.

Figures (13)

  • Figure 1: (a) General RATS in risk-averse decision-making. The pipeline involves amortized evaluation of task difficulties, robust subset selection, policy optimization in the MDP batch, and risk predictive models' update. [fire: updates; snow: evaluation] (b) PDTS as a RATS method. PDTS treats task subsets as bandit arms, evaluates values through posterior sampling, and solves a regularized problem.
  • Figure 2: Task Robust Episodic Learning as a Task-Selection MDP.
  • Figure 3: MPTS's Performance Collapse with Greater $\hat{\mathcal{B}}$. We report the performance collapses of MPTS on Walker2dVel in the case $\hat{\mathcal{B}}=8\mathcal{B}$. The task sampling frequency reveals the presence of the concentration issue.
  • Figure 4: Meta-RL Results. The top depicts the cumulative return curves for $\text{CVaR}_{0.9}$ validation MDPs during meta-training; the middle shows the average cumulative returns curves during meta-training; and the bottom presents the meta-testing results with various $\alpha$.
  • Figure 5: Physical Robotics DR Results. (a) The top shows the cumulative return curves for $\text{CVaR}_{0.9}$ validation MDPs during training; the middle displays the average cumulative return curves across all validation MDPs during training; and the bottom presents the test results at various $\text{CVaR}_{\alpha}$. (b) We evaluate the trained policies in both in-distribution (ID) and out-of-distribution (OOD) domains on LunarLander, reporting the average returns for each sampled task.
  • ...and 8 more figures

Theorems & Definitions (10)

  • Remark 3.1: Bellman Optimality
  • Proposition 3.2: MPTS as a UCB-guided Solution to i-MABs
  • Proposition 3.3: Concentration Issue in Average Top-$\mathcal{B}$ Selection
  • Proposition 3.4: Nearly Worst-Case Optimization with PDTS
  • Definition 1.1: Identifier-Induced MDP Distribution
  • Definition 1.2: $\text{CVaR}_{\alpha}$
  • Lemma 2.1: Subset with Top-$\mathcal{B}$ Risk Values as the Optimal Arm
  • proof
  • proof
  • proof