Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross Domain Policy Transfer with Effect Cycle-Consistency

Ruiqi Zhu, Tianhong Dai, Oya Celiktutan

TL;DR

This paper tackles sample-inefficient deep reinforcement learning by enabling policy transfer across domains with different state and action spaces. It introduces state and action mappings learned from unpaired data using effect cycle-consistency, augmented by a symmetrical optimization framework and inverse-dynamics guidance to align the effects of transitions rather than relying solely on next-state predictions. The resulting objective combines adversarial, cycle-consistency, and effect-consistency losses, and is optimized in an alternating fashion to prevent trivial solutions. Empirical results on 3 locomotion tasks and 2 robotic manipulation tasks show reduced alignment errors and superior transfer performance compared to baselines like Cycle-GAN and Dynamics Cycle-Consistency, validating the practical value of transferring policies across heterogeneous robots without task-specific paired data.

Abstract

Training a robotic policy from scratch using deep reinforcement learning methods can be prohibitively expensive due to sample inefficiency. To address this challenge, transferring policies trained in the source domain to the target domain becomes an attractive paradigm. Previous research has typically focused on domains with similar state and action spaces but differing in other aspects. In this paper, our primary focus lies in domains with different state and action spaces, which has broader practical implications, i.e. transfer the policy from robot A to robot B. Unlike prior methods that rely on paired data, we propose a novel approach for learning the mapping functions between state and action spaces across domains using unpaired data. We propose effect cycle consistency, which aligns the effects of transitions across two domains through a symmetrical optimization structure for learning these mapping functions. Once the mapping functions are learned, we can seamlessly transfer the policy from the source domain to the target domain. Our approach has been tested on three locomotion tasks and two robotic manipulation tasks. The empirical results demonstrate that our method can reduce alignment errors significantly and achieve better performance compared to the state-of-the-art method.

Cross Domain Policy Transfer with Effect Cycle-Consistency

TL;DR

This paper tackles sample-inefficient deep reinforcement learning by enabling policy transfer across domains with different state and action spaces. It introduces state and action mappings learned from unpaired data using effect cycle-consistency, augmented by a symmetrical optimization framework and inverse-dynamics guidance to align the effects of transitions rather than relying solely on next-state predictions. The resulting objective combines adversarial, cycle-consistency, and effect-consistency losses, and is optimized in an alternating fashion to prevent trivial solutions. Empirical results on 3 locomotion tasks and 2 robotic manipulation tasks show reduced alignment errors and superior transfer performance compared to baselines like Cycle-GAN and Dynamics Cycle-Consistency, validating the practical value of transferring policies across heterogeneous robots without task-specific paired data.

Abstract

Training a robotic policy from scratch using deep reinforcement learning methods can be prohibitively expensive due to sample inefficiency. To address this challenge, transferring policies trained in the source domain to the target domain becomes an attractive paradigm. Previous research has typically focused on domains with similar state and action spaces but differing in other aspects. In this paper, our primary focus lies in domains with different state and action spaces, which has broader practical implications, i.e. transfer the policy from robot A to robot B. Unlike prior methods that rely on paired data, we propose a novel approach for learning the mapping functions between state and action spaces across domains using unpaired data. We propose effect cycle consistency, which aligns the effects of transitions across two domains through a symmetrical optimization structure for learning these mapping functions. Once the mapping functions are learned, we can seamlessly transfer the policy from the source domain to the target domain. Our approach has been tested on three locomotion tasks and two robotic manipulation tasks. The empirical results demonstrate that our method can reduce alignment errors significantly and achieve better performance compared to the state-of-the-art method.
Paper Structure (15 sections, 10 equations, 4 figures, 3 tables, 2 algorithms)

This paper contains 15 sections, 10 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background
  4. Effect Cycle-Consistency with Symmetrical Optimization
  5. Experiment
  6. Experiment Setup
  7. The efficacy of our proposed method
  8. Alignment error evaluation
  9. The impact of different dataset sizes
  10. Experiment Results
  11. The efficacy of our proposed method
  12. The alignment error analysis
  13. The impact of different dataset sizes
  14. Conclusion
  15. Acknowledgement

Figures (4)

  • Figure 1: Illustration of the effect cycle-consistency. We aim to align the effect of the transitions across domains. For instance, the effect of transition in Domain A is moving the gripper from the left side to the right side. The effect of the translated transition in the Domain B is expected to move the gripper from the left side to the right side as well.
  • Figure 2: Visulization of the agents in the source domains and the target domains.
  • Figure 3: Illustration of our method.
  • Figure 4: The alignment errors v.s. time steps. (a) HalfCheetah to HalfCheetah 3 legs; (b) Ant to Ant 5 legs.