Adaptive Network Intervention for Complex Systems: A Hierarchical Graph Reinforcement Learning Approach

TL;DR

This paper introduces a Hierarchical Graph Reinforcement Learning (HGRL) framework that governs complex multi-agent systems through targeted interventions in the network structure, and highlights the critical influence of agent-to-agent learning (social learning) on system behavior.

Abstract

Effective governance and steering of behavior in complex multi-agent systems (MAS) are essential for managing system-wide outcomes, particularly in environments where interactions are structured by dynamic networks. In many applications, the goal is to promote pro-social behavior among agents, where network structure plays a pivotal role in shaping these interactions. This paper introduces a Hierarchical Graph Reinforcement Learning (HGRL) framework that governs such systems through targeted interventions in the network structure. Operating within the constraints of limited managerial authority, the HGRL framework demonstrates superior performance across a range of environmental conditions, outperforming established baseline methods. Our findings highlight the critical influence of agent-to-agent learning (social learning) on system behavior: under low social learning, the HGRL manager preserves cooperation, forming robust core-periphery networks dominated by cooperators. In contrast, high social learning accelerates defection, leading to sparser, chain-like networks. Additionally, the study underscores the importance of the system manager's authority level in preventing system-wide failures, such as agent rebellion or collapse, positioning HGRL as a powerful tool for dynamic network-based governance.

Figures (6)

• Figure 1: The diagram of the general process in the environment. We have 3 entities in the whole process: HGRL framework (details are shown in Figure \ref{['fig:mesh1']}), autonomous agents and network topology. In each round of the game, agents with different types interact with their first-order neighbors in the networks; the HGRL framework will observe the information from agents and network topology to make decision of network intervention by adding or deleting links in the network; agents will then imitate others with higher utilities under different imitation probabilities. This process will be iterated for several time steps and we can generate analysis of performance and network evolution from different perspectives.
• Figure 2: General framework of HGRL. HGRL has two agents: node agent and link agent. The node agent will collect graph information using GNNs and select one node to intervene; the link agent will rely on the information related to the selected node to decide to add or delete links to this node. The action spaces for node agent and link agent are O(N), where N is the number of nodes in the network.
• Figure 3: The performance comparison of HGRL, Flat-RL and random strategy in the system with 10 nodes. The figures show results from 3 scenarios with different probabilities of behavior imitation. Average social welfare (red bar) during the game and average final social welfare (blue bar) at the end of the game are used as metrics.
• Figure 4: The performance comparison of HGRL, Flat-RL and random strategy in the system with 20 nodes. The figures show results from 3 scenarios with different probabilities of behavior imitation. Average social welfare (red bar) during the game and average final social welfare (blue bar) at the end of the game are used as metrics.
• Figure 5: The evolution of different network metrics over time with various imitation probabilities. The figures show 3 network metrics: average node degree, network diameter, and network modularity. The x-axis is the time step.