Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning

Zhangjie Xia, Yu Yang, Pan Xu

TL;DR

Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch and avoids overly coarse global assumptions and expensive per-sample filtering.

Abstract

Off-dynamics offline reinforcement learning (RL) aims to learn a policy for a target domain using limited target data and abundant source data collected under different transition dynamics. Existing methods typically address dynamics mismatch either globally over the state space or via pointwise data filtering; these approaches can miss localized cross-domain similarities or incur high computational cost. We propose Localized Dynamics-Aware Domain Adaptation (LoDADA), which exploits localized dynamics mismatch to better reuse source data. LoDADA clusters transitions from source and target datasets and estimates cluster-level dynamics discrepancy via domain discrimination. Source transitions from clusters with small discrepancy are retained, while those from clusters with large discrepancy are filtered out. This yields a fine-grained and scalable data selection strategy that avoids overly coarse global assumptions and expensive per-sample filtering. We provide theoretical insights and extensive experiments across environments with diverse global and local dynamics shifts. Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch.

Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning

TL;DR

Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch and avoids overly coarse global assumptions and expensive per-sample filtering.

Abstract

Off-dynamics offline reinforcement learning (RL) aims to learn a policy for a target domain using limited target data and abundant source data collected under different transition dynamics. Existing methods typically address dynamics mismatch either globally over the state space or via pointwise data filtering; these approaches can miss localized cross-domain similarities or incur high computational cost. We propose Localized Dynamics-Aware Domain Adaptation (LoDADA), which exploits localized dynamics mismatch to better reuse source data. LoDADA clusters transitions from source and target datasets and estimates cluster-level dynamics discrepancy via domain discrimination. Source transitions from clusters with small discrepancy are retained, while those from clusters with large discrepancy are filtered out. This yields a fine-grained and scalable data selection strategy that avoids overly coarse global assumptions and expensive per-sample filtering. We provide theoretical insights and extensive experiments across environments with diverse global and local dynamics shifts. Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch.
Paper Structure (41 sections, 5 theorems, 38 equations, 9 figures, 11 tables, 2 algorithms)

This paper contains 41 sections, 5 theorems, 38 equations, 9 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology
  5. Theoretical Insights
  6. Clustering-Based KL Estimation
  7. Policy Optimization
  8. Experiment
  9. Experimental Setup
  10. Results on MuJoCo Locomotion Tasks
  11. Experiments on Localized Perturbations
  12. Results on Antmaze
  13. Ablation study
  14. Conclusion
  15. Proof of the Main Theoretical Results
  16. ...and 26 more sections

Key Result

Proposition 4.1

For any $( s, a )$, denote its representation as $z$, and suppose $s_{\text{src }}^{\prime} \sim P_{\mathcal{M}_{\mathrm{src}}}(\cdot | z), s_{\mathrm{tar}}^{\prime} \sim P_{\mathcal{M}_{\mathrm{tar}}}(\cdot | z)$. Then measuring the representation deviation between the source domain and the target

Figures (9)

  • Figure 1: An overview of our proposed framework. We first perform K-means clustering on the mixed source and target dataset. Then we estimate the local KL divergence between source and target dynamics in each cluster. We use cluster-wise classifiers to selectively retain source domain transitions for downstream offline RL algorithms. We further introduce a dataset regularization term to ensure policy consistency with the target dataset.
  • Figure 2: Parameter sensitivity study.
  • Figure 3: Visualization of morphology shift environments for Ant.
  • Figure 4: Visualization of morphology shift environments for HalfCheetah.
  • Figure 5: Visualization of morphology shift environments for Hopper.
  • ...and 4 more figures

Theorems & Definitions (6)

  • Proposition 4.1
  • Theorem 4.2: Offline performance bound
  • Remark 4.3
  • Lemma 1
  • Lemma 2: Telescoping lemma, Lemma 4.3 from luo2021algorithmicframeworkmodelbaseddeep
  • Lemma 3