Contextual Knowledge Sharing in Multi-Agent Reinforcement Learning with Decentralized Communication and Coordination

TL;DR

A novel Dec-MARL framework that integrates peer-to-peer communication and coordination, incorporating goal-awareness and time-awareness into the agents' knowledge-sharing processes is presented, demonstrating that incorporating goal-aware and time-aware knowledge sharing significantly enhances overall performance.

Abstract

Decentralized Multi-Agent Reinforcement Learning (Dec-MARL) has emerged as a pivotal approach for addressing complex tasks in dynamic environments. Existing Multi-Agent Reinforcement Learning (MARL) methodologies typically assume a shared objective among agents and rely on centralized control. However, many real-world scenarios feature agents with individual goals and limited observability of other agents, complicating coordination and hindering adaptability. Existing Dec-MARL strategies prioritize either communication or coordination, lacking an integrated approach that leverages both. This paper presents a novel Dec-MARL framework that integrates peer-to-peer communication and coordination, incorporating goal-awareness and time-awareness into the agents' knowledge-sharing processes. Our framework equips agents with the ability to (i) share contextually relevant knowledge to assist other agents, and (ii) reason based on information acquired from multiple agents, while considering their own goals and the temporal context of prior knowledge. We evaluate our approach through several complex multi-agent tasks in environments with dynamically appearing obstacles. Our work demonstrates that incorporating goal-aware and time-aware knowledge sharing significantly enhances overall performance.

Figures (5)

• Figure 1: An illustration of a fully decentralized environment with multiple agents ($t < t'$). While the goal of Agents 1 and 2 is G1, that of Agent 3 is G2. Note that at time $t'$, Agent A3 is unaware of an obstacle that has occurred in a position it previously encountered, rendering its knowledge about that location obsolete. Additionally, during a communication session with A3, Agents A1 and A2 must be aware of the outdated information provided by A3 to select the optimal action.
• Figure 2: A demonstration of the intrinsic reward guiding Agent 1 (A1) to choose an action that optimizes both the goal-oriented objective and the exploration of uncertainty. The filled yellow boxes represent knowledge of A1 in terms of that position, the red box filled by dots is obstacle, and the remaining are unknown to A1. In this scenario, the optimal action for A1 is to move towards the $(4, 2)$ position, as it strikes a balance between both objectives.
• Figure 3: An illustration of utilizing Equation \ref{['eq:time-nov']} to estimate the novelty of information over 100 steps where $d_{t'}$ is estimated with the time increment of 0.01 as: $t' = t + 0.01$. Importantly, in this graph, we assume that the information is not reflected by an agent per step.
• Figure 4: An illustration of the Base environment
• Figure 5: An illustration of the Large environment