Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations

TL;DR

This paper addresses how groups of reinforcement-learning agents can master complex policies from scratch by exchanging action recommendations, without relying on expert demonstrations. It formalizes trust in advice as a non-stationary multi-armed bandit problem and introduces multiple weighting mechanisms (Critic, Local Trust, Agent Values) plus an Advantage variant to assess and follow peers. Empirically, peer learning improves learning speed and final performance over single-agent baselines and early-advising baselines in MuJoCo continuous-control tasks, while robust trust mechanisms help identify and ignore adversarial peers. The work suggests practical potential for cross-institutional, parallel RL training and outlines future directions for scaling to larger agent groups and deeper theoretical analysis of social learning in RL.

Abstract

Peer learning is a novel high-level reinforcement learning framework for agents learning in groups. While standard reinforcement learning trains an individual agent in trial-and-error fashion, all on its own, peer learning addresses a related setting in which a group of agents, i.e., peers, learns to master a task simultaneously together from scratch. Peers are allowed to communicate only about their own states and actions recommended by others: "What would you do in my situation?". Our motivation is to study the learning behavior of these agents. We formalize the teacher selection process in the action advice setting as a multi-armed bandit problem and therefore highlight the need for exploration. Eventually, we analyze the learning behavior of the peers and observe their ability to rank the agents' performance within the study group and understand which agents give reliable advice. Further, we compare peer learning with single agent learning and a state-of-the-art action advice baseline. We show that peer learning is able to outperform single-agent learning and the baseline in several challenging discrete and continuous OpenAI Gym domains. Doing so, we also show that within such a framework complex policies from action recommendations beyond discrete action spaces can evolve.

Figures (7)

• Figure 1: Peer learning: Agent $2$ (right) asks for advice about which action to perform at the current step $S_{t, 2}$ it finds itself in, and receives action suggestions $A^{0}_{t, 2}$ and $A^{1}_{t, 2}$ from agents $0$ and $1$, on the left and center, respectively. The agents operate in their own environment and on their own complex (motor) task, here: learning how to handle a ball.
• Figure 2: An intricate depiction of the advice exchange in peer learning with three agents. Agent $0$ operates within its environment and requests advice in state $S_{t,0}$ from all peers in the group, including itself (outlined in dotted frame). In return, it receives suggested actions $A_{t, 0}^0, \dots, A_{t, 0}^2$ and then selects an action from those suggestions according to its peer policy $\Pi_0$.
• Figure 3: Our Room environment of size $5 \times 5$. The agent (robot) starts in the center of a square grid-world, where the goal position is a random position at the border (flag). It can choose among the four actions: right, left, up and down.
• Figure 4: The reliability mechanisms in our approach protect the learning against the influence of a malicious agent. Peer learning cannot be harmed by an adversarial agent and still outperforms single agent learning.
• Figure 5: In a comparison, our approach (Peer Learning) outperforms the LeCTR approach omidshafiei2019 and a solo RL agent (Single Agent) on average over 10 random seeds on the Room environment of size $21 \times 21$. The shaded area marks the standard error of the mean.