Dynamic Evolution of Complex Networks: A Reinforcement Learning Approach Applying Evolutionary Games to Community Structure

TL;DR

This work presents a dynamic network framework that integrates birth-death processes, reinforcement-learning–driven games, and a 2D spatial embedding to study the emergence and evolution of community structures. Agents move on a lattice using Q-learning, form and decay interaction edges, and compete in a snowdrift game, with strategy updates governed by a Fermi rule. The model yields theoretical results for the stationary distribution of population size and demonstrates strong alignment with real data across both population dynamics and network degree distributions. Key findings show that exploitation rates, payoff parameters, learning rate, discount factor, and spatial dimensionality shape cooperation and community formation, while the birth-death process can modulate the pace and extent of clustering. The framework achieves good empirical fits to multiple countries and real networks, highlighting its practical relevance for understanding population dynamics, network evolution, and community structure formation in complex systems.

Abstract

Complex networks serve as abstract models for understanding real-world complex systems and provide frameworks for studying structured dynamical systems. This article addresses limitations in current studies on the exploration of individual birth-death and the development of community structures within dynamic systems. To bridge this gap, we propose a networked evolution model that includes the birth and death of individuals, incorporating reinforcement learning through games among individuals. Each individual has a lifespan following an arbitrary distribution, engages in games with network neighbors, selects actions using Q-learning in reinforcement learning, and moves within a two-dimensional space. The developed theories are validated through extensive experiments. Besides, we observe the evolution of cooperative behaviors and community structures in systems both with and without the birth-death process. The fitting of real-world populations and networks demonstrates the practicality of our model. Furthermore, comprehensive analyses of the model reveal that exploitation rates and payoff parameters determine the emergence of communities, learning rates affect the speed of community formation, discount factors influence stability, and two-dimensional space dimensions dictate community size. Our model offers a novel perspective on real-world community development and provides a valuable framework for studying population dynamics behaviors.

Figures (13)

• Figure 1: An example of the network structure generated according to our model. The figure shows a network structure generated based on our model, where different colors denote different community structures, suggesting that the model proposed in this paper can provide an explanation for the formation and development of community structures in reality.
• Figure 2: Phase transition of system scale. The system scale undergoes a transition from $k - 1$ to $k$ at an exponential rate of $\lambda_{k-1}$ or from $k$ to $k - 1$ at a general rate of $\mu_k$, and the system size is constrained within the range of $[0, \infty)$.
• Figure 3: An illustration of the model. The figure shows the evolution example of the complex network model. Black duration indicates the lifetime of the individual, which follows a general distribution. The individual will be removed from the system once its lifetime ends. We select $t_1$ and $t_2$ to observe the snapshots of the two-dimensional space and the network, which have a one-to-one correspondence between them. When $t=t_2$, individuals B and E in $t_1$ have died since their lifetimes are over, and new individuals G, H, and I are born in the system. The connections between newly born individuals and individuals already in the network are shown in green.
• Figure 4: Evolutionary curves and statistical distributions of the number of individuals. In this figure, we show both the (a) evolutionary curves and (b) statistical distribution of the individual number in the system with the birth-death process under various death processes, including power-law, uniform, exponential, and lognormal distributions. In addition, we utilize different markers and colors for comparison among different death processes. The red line indicates the theoretical value for each specific death process.
• Figure 5: Heat maps of cooperation fraction about payoff parameter $r$ and exploitation rate $\delta$. By setting the $y$-axis as the exploitation rate $\delta$ with a range [0, 1] and the $x$-axis as the payoff parameter $r$ with a range [0, 1], we demonstrate the heat maps of cooperation fraction on the system with (in panel (a)) and without (in panel (b)) birth-death process with respect to payoff parameter $r$ and exploitation rate $\delta$. The learning rate, discount factor, and weight fading factor of the edge in both subfigures are all set to 0.7, 0.3, and 2.

Theorems & Definitions (2)

• Theorem 1
• Theorem 2