Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, Ruslan Salakhutdinov

TL;DR

This work tackles transfer learning in reinforcement learning under dynamics mismatch by framing domain adaptation as reward shaping guided by a variational objective. The proposed DARC method learns two domain classifiers to estimate a Delta r reward correction and integrates it with a max-entropy RL algorithm to produce near-optimal policies in the target domain. Theoretical guarantees accompany empirical results showing improved sample efficiency across discrete and continuous control tasks, including safety-related termination differences, with ablations underscoring the importance of dual classifiers and regularization. Overall, DARC offers a practical, model-free approach to robust transfer across domains with differing dynamics.

Abstract

We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain. Said another way, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional tasks.

Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

TL;DR

This work tackles transfer learning in reinforcement learning under dynamics mismatch by framing domain adaptation as reward shaping guided by a variational objective. The proposed DARC method learns two domain classifiers to estimate a Delta r reward correction and integrates it with a max-entropy RL algorithm to produce near-optimal policies in the target domain. Theoretical guarantees accompany empirical results showing improved sample efficiency across discrete and continuous control tasks, including safety-related termination differences, with ablations underscoring the importance of dual classifiers and regularization. Overall, DARC offers a practical, model-free approach to robust transfer across domains with differing dynamics.

Abstract

We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain. Said another way, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional tasks.

Paper Structure

This paper contains 38 sections, 4 theorems, 28 equations, 13 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. A Variational Perspective on Domain Adaptation in RL
  5. Theoretical Guarantees
  6. Domain Adaptation in RL with a Learned Reward
  7. Algorithm Summary
  8. Experiments
  9. Illustrative example.
  10. Visualizing the reward modification in stochastic domains.
  11. Scaling to more complex tasks.
  12. Ablation Experiments.
  13. Safety emerges from domain adaptation to the termination condition.
  14. Comparison with Prior Transfer Learning Methods.
  15. Discussion
  16. ...and 23 more sections

Key Result

Theorem 4.1

Let $\pi_\text{DARC}^*$ be the policy that maximizes the modified (entropy-regularized) reward in the source domain, let $\pi^*$ be the policy that maximizes the (unmodified, entropy-regularized) reward in the target domain, and assume that $\pi^*$ satisfies Assumption assumption:opt. Then the follo

Figures (13)

  • Figure 1: Our method acquires a policy for the target domain by practicing in the source domain using a (learned) modified reward function.
  • Figure 2: Block diagram of DARC (Alg. \ref{['alg:odrl']})
  • Figure 3: Tabular example of off-dynamics RL
  • Figure 4: Visualizing the modified reward
  • Figure 5: Environments: broken reacher, broken half cheetah, broken ant, and half cheetah obstacle.
  • ...and 8 more figures

Theorems & Definitions (7)

  • Definition 1
  • Theorem 4.1
  • Lemma B.1
  • Lemma B.2
  • proof
  • Theorem 4.1: Repeated from main text
  • proof