DRAMA: A Dynamic Packet Routing Algorithm using Multi-Agent Reinforcement Learning with Emergent Communication

TL;DR

DRAMA addresses dynamic packet routing by formulating routing as a Dec-POMDP and introducing emergent communication within a MARL framework. It combines an observation-encoding layer, a graph-based emergent communication layer with attention, and a neighbor-aware Q-network to evaluate per-neighbor actions, trained with TD and an estimated-cost constraint. Across static, dynamic, and real-world ATT topologies, DRAMA achieves superior delivery rates and lower latency, and can adapt to router additions or failures without retraining. The work demonstrates that emergent communication enables self-organized collaboration among routers, improving scalability and robustness for real-time network routing.

Abstract

The continuous expansion of network data presents a pressing challenge for conventional routing algorithms. As the demand escalates, these algorithms are struggling to cope. In this context, reinforcement learning (RL) and multi-agent reinforcement learning (MARL) algorithms emerge as promising solutions. However, the urgency and importance of the problem are clear, as existing RL/MARL-based routing approaches lack effective communication in run time among routers, making it challenging for individual routers to adapt to complex and dynamic changing networks. More importantly, they lack the ability to deal with dynamically changing network topology, especially the addition of the router, due to the non-scalability of their neural networks. This paper proposes a novel dynamic routing algorithm, DRAMA, incorporating emergent communication in multi-agent reinforcement learning. Through emergent communication, routers could learn how to communicate effectively to maximize the optimization objectives. Meanwhile, a new Q-network and graph-based emergent communication are introduced to dynamically adapt to the changing network topology without retraining while ensuring robust performance. Experimental results showcase DRAMA's superior performance over the traditional routing algorithm and other RL/MARL-based algorithms, achieving a higher delivery rate and lower latency in diverse network scenarios, including dynamic network load and topology. Moreover, an ablation experiment validates the prospect of emergent communication in facilitating packet routing.

Figures (7)

• Figure 1: Overview of DRAMA
• Figure 2: Two types of traditional Q-network and our proposed method. Router 0 calculates the Q-value for each neighbor. It has the original neighbors {1,2,3}, and the new neighbor 4. $o_i, a_i, f_i$ are observation, action, and feature of agent $i$, respectively, and we assume enumerative $a_i=i$.
• Figure 3: Network Topology Used In Simulator.
• Figure 4: Ablation Experiments
• Figure 5: Results of Communication Round Experiments