Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review

TL;DR

A comprehensive and systematic revision of knowledge transfer methods for the cross-domain RL setting, a categorization and characterization of such methods to provide an analysis based on relevant features such as their transfer approach and data requirements, and a discussion on the main challenges regarding cross-domain knowledge transfer are presented.

Abstract

Reinforcement Learning (RL) provides a framework in which agents can be trained, via trial and error, to solve complex decision-making problems. Learning with little supervision causes RL methods to require large amounts of data, rendering them too expensive for many applications (e.g., robotics). By reusing knowledge from a different task, knowledge transfer methods present an alternative to reduce the training time in RL. Given the severe data scarcity, due to their flexibility, there has been a growing interest in methods capable of transferring knowledge across different domains (i.e., problems with different representations). However, identifying similarities and adapting knowledge across tasks from different domains requires matching their representations or finding domain-invariant features. These processes can be data-demanding, which poses the main challenge in cross-domain knowledge transfer: to select and transform knowledge in a data-efficient way, such that it accelerates learning in the target task, despite the presence of significant differences across problems (e.g., robots with distinct morphologies). Thus, this review presents a unifying analysis of methods focused on transferring knowledge across different domains. Through a taxonomy based on a transfer-approach categorization and a characterization of works based on their data-assumption requirements, the contributions of this article are 1) a comprehensive and systematic revision of knowledge transfer methods for the cross-domain RL setting, 2) a categorization and characterization of such methods to provide an analysis based on relevant features such as their transfer approach and data requirements, and 3) a discussion on the main challenges regarding cross-domain knowledge transfer, as well as on ideas of future directions worth exploring to address these problems.

Figures (3)

• Figure 1: This review covers works that study transferring knowledge across single-agent reinforcement learning domains that have different state and/or action representations.
• Figure 2: The difference between an MDP and POMDP setting is that the MDP agent has access to the system's state, whereas the POMDP agent perceives observations that depend on the system's state but do not fully describe it. The POMDP agent must learn a policy based on a constantly updated state estimation.
• Figure 3: Transfer learning metrics evaluate the effect of transferring knowledge by measuring the performance difference, whether it is the initial performance (jumpstart), final performance (asymptotic), or the time it took to achieve certain performance (time to threshold).

Theorems & Definitions (4)

• Definition 3.1: Domain
• Definition 3.2: Task
• Definition 3.3: Space Mismatch
• Definition 3.4: Cross-Domain Knowledge Transfer