Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transfer in Reinforcement Learning via Regret Bounds for Learning Agents

Adrienne Tuynman, Ronald Ortner

TL;DR

This work tackles the problem of quantifying the usefulness of transfer in reinforcement learning by introducing mutual regret for a multi-agent MARL setting where agents share observations in a common environment. It adapts the UCRL2 framework into Multi-agent-UCRL, deriving a high-probability bound on the mutual regret that scales with $\sqrt{SA\aleph T\log(8A\aleph T/\delta)}$ and includes a $\sqrt{\aleph}$ term, implying per-agent gains of $\tilde{O}(DS\sqrt{AT}/\sqrt{\aleph})$ under certain conditions. The key contributions are (i) the first regret-based theoretical guarantees for simultaneous multi-agent RL with transfer, (ii) a concrete bound showing that sharing observations reduces joint regret by a factor related to the number of agents, and (iii) empirical validation on RiverSwim and SixArms confirming the practical benefit of mutual transfer. These results provide a principled, quantitative measure of transfer utility and suggest that observation sharing can substantially improve learning efficiency in multi-task RL scenarios.

Abstract

We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of $\aleph$ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing her average reward. We show that when the agents share their observations the total regret of all agents is smaller by a factor of $\sqrt{\aleph}$ compared to the case when each agent has to rely on the information collected by herself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.

Transfer in Reinforcement Learning via Regret Bounds for Learning Agents

TL;DR

This work tackles the problem of quantifying the usefulness of transfer in reinforcement learning by introducing mutual regret for a multi-agent MARL setting where agents share observations in a common environment. It adapts the UCRL2 framework into Multi-agent-UCRL, deriving a high-probability bound on the mutual regret that scales with and includes a term, implying per-agent gains of under certain conditions. The key contributions are (i) the first regret-based theoretical guarantees for simultaneous multi-agent RL with transfer, (ii) a concrete bound showing that sharing observations reduces joint regret by a factor related to the number of agents, and (iii) empirical validation on RiverSwim and SixArms confirming the practical benefit of mutual transfer. These results provide a principled, quantitative measure of transfer utility and suggest that observation sharing can substantially improve learning efficiency in multi-task RL scenarios.

Abstract

We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing her average reward. We show that when the agents share their observations the total regret of all agents is smaller by a factor of compared to the case when each agent has to rely on the information collected by herself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.
Paper Structure (20 sections, 3 theorems, 21 equations, 2 figures, 1 algorithm)

This paper contains 20 sections, 3 theorems, 21 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Mutual Regret in a Multi-Agent Setting
  3. Related Work
  4. Contributions and Outline
  5. MDP Preliminaries
  6. Assumptions and the Diameter
  7. The Learning Goal and the Notion of Regret
  8. Transfer Learning
  9. Sequential Transfer Learning
  10. Mutual Transfer and Regret in a Multi-Agent Setting
  11. Regret Bounds for Multi-Agent Learning
  12. UCRL2 with Shared Information
  13. Upper Bound on the Mutual Regret
  14. Discussion
  15. Numerical experiments
  16. ...and 5 more sections

Key Result

Theorem 1

With probability $1-\delta$, after any $T$ steps the mutual regret of agents controlled by Multi-agent-UCRL with confidence intervals eq:cr and eq:cp is upper bounded by

Figures (2)

  • Figure 1: An experiment in the SixArm environment, with (left) the average mutual regret for $\aleph=1$ (UCRL2) and $\aleph=3,6,10,15$ over 64 runs, and (right) the regret of each of the 15 agents of Multi-agent-UCRL with shared information in one run.
  • Figure 2: An experiment in the RiverSwim environment, with (left) the average mutual regret for $\aleph=1$ (UCRL2) and $\aleph=3,6,10,15$ over 64 runs, and (right) the regret of each of the 15 agents in one run of Multi-agent-UCRL with shared information.

Theorems & Definitions (5)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Corollary 1
  • Theorem 2