Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structure Detection for Contextual Reinforcement Learning

Tianyue Zhou, Jung-Hoon Cho, Cathy Wu

TL;DR

The paper tackles CRL in CMDPs by introducing Structure Detection MBTL (SD-MBTL), a framework that infers the underlying generalization structure from observed transfer performance and selects an appropriate MBTL algorithm accordingly. Its instantiation, M/GP-MBTL, switches between Mountain-structured clustering (M-MBTL) and Gaussian Process MBTL (GP-MBTL) to balance sample efficiency and robustness. Across synthetic and real CMDP benchmarks, including CartPole, BipedalWalker, IntersectionZoo, and CyclesGym, M/GP-MBTL achieves the strongest aggregated performance, outperforming prior MBTL methods by up to 12.49% and closely approaching the Myopic Oracle. The work highlights online structure detection as a promising direction for scalable, robust source-task selection in complex CRL environments, while noting limitations and avenues for extending to richer CMDP structures and higher-dimensional contexts.

Abstract

Contextual Reinforcement Learning (CRL) tackles the problem of solving a set of related Contextual Markov Decision Processes (CMDPs) that vary across different context variables. Traditional approaches--independent training and multi-task learning--struggle with either excessive computational costs or negative transfer. A recently proposed multi-policy approach, Model-Based Transfer Learning (MBTL), has demonstrated effectiveness by strategically selecting a few tasks to train and zero-shot transfer. However, CMDPs encompass a wide range of problems, exhibiting structural properties that vary from problem to problem. As such, different task selection strategies are suitable for different CMDPs. In this work, we introduce Structure Detection MBTL (SD-MBTL), a generic framework that dynamically identifies the underlying generalization structure of CMDP and selects an appropriate MBTL algorithm. For instance, we observe Mountain structure in which generalization performance degrades from the training performance of the target task as the context difference increases. We thus propose M/GP-MBTL, which detects the structure and adaptively switches between a Gaussian Process-based approach and a clustering-based approach. Extensive experiments on synthetic data and CRL benchmarks--covering continuous control, traffic control, and agricultural management--show that M/GP-MBTL surpasses the strongest prior method by 12.49% on the aggregated metric. These results highlight the promise of online structure detection for guiding source task selection in complex CRL environments.

Structure Detection for Contextual Reinforcement Learning

TL;DR

The paper tackles CRL in CMDPs by introducing Structure Detection MBTL (SD-MBTL), a framework that infers the underlying generalization structure from observed transfer performance and selects an appropriate MBTL algorithm accordingly. Its instantiation, M/GP-MBTL, switches between Mountain-structured clustering (M-MBTL) and Gaussian Process MBTL (GP-MBTL) to balance sample efficiency and robustness. Across synthetic and real CMDP benchmarks, including CartPole, BipedalWalker, IntersectionZoo, and CyclesGym, M/GP-MBTL achieves the strongest aggregated performance, outperforming prior MBTL methods by up to 12.49% and closely approaching the Myopic Oracle. The work highlights online structure detection as a promising direction for scalable, robust source-task selection in complex CRL environments, while noting limitations and avenues for extending to richer CMDP structures and higher-dimensional contexts.

Abstract

Contextual Reinforcement Learning (CRL) tackles the problem of solving a set of related Contextual Markov Decision Processes (CMDPs) that vary across different context variables. Traditional approaches--independent training and multi-task learning--struggle with either excessive computational costs or negative transfer. A recently proposed multi-policy approach, Model-Based Transfer Learning (MBTL), has demonstrated effectiveness by strategically selecting a few tasks to train and zero-shot transfer. However, CMDPs encompass a wide range of problems, exhibiting structural properties that vary from problem to problem. As such, different task selection strategies are suitable for different CMDPs. In this work, we introduce Structure Detection MBTL (SD-MBTL), a generic framework that dynamically identifies the underlying generalization structure of CMDP and selects an appropriate MBTL algorithm. For instance, we observe Mountain structure in which generalization performance degrades from the training performance of the target task as the context difference increases. We thus propose M/GP-MBTL, which detects the structure and adaptively switches between a Gaussian Process-based approach and a clustering-based approach. Extensive experiments on synthetic data and CRL benchmarks--covering continuous control, traffic control, and agricultural management--show that M/GP-MBTL surpasses the strongest prior method by 12.49% on the aggregated metric. These results highlight the promise of online structure detection for guiding source task selection in complex CRL environments.
Paper Structure (66 sections, 3 theorems, 22 equations, 23 figures, 10 tables, 7 algorithms)

This paper contains 66 sections, 3 theorems, 22 equations, 23 figures, 10 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Contextual Markov Decision Process
  4. Problem Formulation
  5. Model-Based Transfer Learning
  6. Structure Detection Model-Based Transfer Learning
  7. Generalization Performance Structure Decomposition
  8. SD-MBTL
  9. M/GP Model-Based Transfer Learning
  10. Mountain Structure
  11. Structure Detection
  12. Mountain Model-Based Transfer Learning
  13. M/GP-MBTL
  14. Experiments
  15. Setup
  16. ...and 51 more sections

Key Result

Lemma 4.4

If $\text{std}_{x\in x_{1:k}}(\overline{J}(\pi_x,x))<\mathbb{E}_{x\in x_{1:k}}[\text{std}_{y\in Y}(\overline{J}(\pi_x,y))]$, then we have$\textcolor{black}{std_{x\in x_{1:k}}(f(x))<\mathbb{E}_{x\in x_{1:k}}[std_{y\in Y} (h(x,y))].}$

Figures (23)

  • Figure 1: Conceptual overview.SD-MBTL detects an underlying structure from observed generalization performance and selects an appropriate algorithm.
  • Figure 2: Heatmap of policy quality $f(x)$, task difficulty $g(y)$, and task dissimilarity $h(x,y)$ for CartPole (mass of cart), BipedalWalker (scale), and CyclesGym (precipitation) averaged over three different random seeds. In these tasks, $f(x)$ is nearly constant. $h(x, y)$ decreases approximately linearly as the context difference increases, resembling a distance metric.
  • Figure 3: Overview illustration for M-MBTL. (a) A distance metric is used to estimate the generalization gaps between tasks and existing policies. (b) A random restart approach is employed to search for the next training task. $M$ tasks are sampled as initial candidates, with each initial candidate serving as a new centroid candidate while previous centroids are fixed. The candidate is optimized based on the clustering loss, and the optimized candidate with the minimum loss is selected as the next training task. (c) Train on the selected task.
  • Figure 4: CPU time comparison of M/GP-MBTL with M-MBTL and GP-MBTL on the CartPole, BipedalWalker, IntersectionZoo, and CyclesGym benchmarks.
  • Figure 5: Transfer matrices of three context variables in the CartPole task (pole length, cart mass, pole mass). Brighter colors indicate highergeneralization performance.
  • ...and 18 more figures

Theorems & Definitions (10)

  • Definition 3.1: Generalization Performance Structure Decomposition
  • Definition 4.3: Mountain Structure
  • Lemma 4.4: Relative Influence of Policy Quality and Task Dissimilarity
  • Lemma 4.5: Reduction of GSTS to Clustering
  • Theorem C.1: Sobol–Hoeffding decomposition for two variables hoffding1948classsobol1990sensitivity
  • proof
  • proof
  • proof
  • Definition L.1: Myopic Oracle
  • Definition L.2: Sequential Oracle