Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning
Alberto Castagna
TL;DR
This work introduces EF-OnTL, an online, expert-free transfer learning framework for multi-agent reinforcement learning that enables dynamic, uncertainty-guided sharing of agent-environment experiences among homogeneous agents. Central to EF-OnTL is sarnd, an enhanced uncertainty estimator that extends RND by incorporating action, reward, and next state to better guide online transfer decisions. The framework features a Transfer Core Engine with two components: source selection (based on agent uncertainty or performance) and transfer content selection (using delta-conf and TD-error-based criteria) to curate batches of experiences for target agents. Empirical evaluation across Cart-Pole, Predator-Prey, Half-Field Offense, and rs-sumo demonstrates that EF-OnTL can accelerate learning and improve performance relative to No-Transfer and action-advice baselines, though results vary with task complexity and can exhibit negative transfer in some settings. Overall, EF-OnTL provides a scalable, decentralized alternative to fixed-expert TL, enhancing learning efficiency in MAS without requiring ongoing expert supervision, and opens pathways for adaptive, data-driven transfer strategies in diverse RL problems.
Abstract
Reinforcement Learning (RL) enables an intelligent agent to optimise its performance in a task by continuously taking action from an observed state and receiving a feedback from the environment in form of rewards. RL typically uses tables or linear approximators to map state-action tuples that maximises the reward. Combining RL with deep neural networks (DRL) significantly increases its scalability and enables it to address more complex problems than before. However, DRL also inherits downsides from both RL and deep learning. Despite DRL improves generalisation across similar state-action pairs when compared to simpler RL policy representations like tabular methods, it still requires the agent to adequately explore the state-action space. Additionally, deep methods require more training data, with the volume of data escalating with the complexity and size of the neural network. As a result, deep RL requires a long time to collect enough agent-environment samples and to successfully learn the underlying policy. Furthermore, often even a slight alteration to the task invalidates any previous acquired knowledge. To address these shortcomings, Transfer Learning (TL) has been introduced, which enables the use of external knowledge from other tasks or agents to enhance a learning process. The goal of TL is to reduce the learning complexity for an agent dealing with an unfamiliar task by simplifying the exploration process. This is achieved by lowering the amount of new information required by its learning model, resulting in a reduced overall convergence time...