Shaping the learning signal in a combined Q-learning rule to improve structured cooperation

TL;DR

The paper investigates how reputation can promote cooperation by shaping the learning signal in Q-learning on a spatial Prisoner’s Dilemma. The method keeps the game and network fixed while using a reward signal $\Pi_i(t)=(1-\beta)\pi_i(t)+\beta R_i(t)$ to update Q-values, enabling explicit analysis of learning dynamics. They find that cooperation generally rises with the reputation weight $\beta$, but the effect vanishes in two regimes: $\alpha$ approaching 0 and $\gamma$ approaching 1; outside these, larger $\alpha$ reduces the benefit while larger $\gamma$ enhances it. This work shows that social information can be harnessed through learning dynamics to stabilize network reciprocity, with implications for reputation-based interventions; future work should explore more complex topologies, second-order reputation, and adaptive learning rates.

Abstract

Q-learning provides a standard reinforcement learning framework for studying cooperation by specifying how agents update action values from repeated local interactions outcomes. Although previous work has shown that reputation can promote cooperation in such systems, most models introduce reputation by modifying payoffs, encoding it directly in the state or changing partner selection, which makes it difficult to isolate the role of the learning signal itself. Here, we construct the reinforcement signal as a weighted combination of reputation and game payoffs, leaving the game and network structure unchanged. We find that increasing the weight on reputation generally promotes cooperation by consolidating clusters, but this effect is conditional on the learning dynamics. Specifically, this promoting effect vanishes in two regimes: when the learning rate is extremely small, which prevents effective information propagation and when the discount factor approaches one, as distant future expectations obscure the immediate reputational advantage. Outside these limiting cases, the efficacy of reputation in promoting cooperation is attenuated by higher learning rates but amplified by larger discount factors. These results advance the understanding of cooperative dynamics by demonstrating that cooperation can be stabilized through the reputational shaping of learning signals alone, providing critical insights into the interplay between social information and individual learning parameters.

Figures (7)

• Figure 1: The frequency of cooperators $\rho_c$ in dependence of the temptation to defect $b$, under various values of the reputation weight $\beta$. The values of $\beta$ range from [0,1] as indicated in the legend. All results are obtained under the parameter settings of $\alpha = 0.1$, $\gamma = 0.95$ and $\epsilon = 0.01$. The figure suggests that a larger reputation weight has a positive consequence on general cooperation.
• Figure 2: The cooperation level on the parameter plane of $\alpha$ and reputation weight (a) and on the parameter plane of discount factor $\gamma$ and reputation weight (b). The color-coded stationary values of $\rho_C$ are indicated by the bar shown on the right-hand side. While the effect of parameter $\beta$ on $\rho_C$ is straightforward, it cannot be said about the effects of other parameters $\alpha$ and $\gamma$. We can identify an intermediate optimal $\alpha$ and $\gamma$ values in both panels where the highest cooperation level can be reached. Other parameters are set as $b = 1.02$ and $\epsilon = 0.01$.
• Figure 3: The time evolution of spatial patterns at different reputation weight $\beta$. From top to bottom, the values of $\beta$ are 0.0, 0.5 and 1.0. The snapshots were taken at time steps $T = 0, 1000, 10000, 50000$ and $99999$. Defectors and cooperators are represented by blue and red cells, respectively. As expected, larger $\beta$ value provides an increased cooperator community. These players, however, do not form as compact clusters as we could previously observed when imitation dynamics was used. The remaining parameters are set as $\alpha = 0.8$, $\gamma = 0.6$, $b = 1.05$ and $\epsilon = 0.01$.
• Figure 4: The time evolution of spatial patterns at different reputation weight $\alpha$. From top to bottom, the values of $\alpha$ are 0.0, 0.1 and 0.3. The snapshots were taken at time steps $T = 0, 1000, 10000, 50000$ and $99999$. Defectors and cooperators are represented by blue and red cells, respectively. The non-monotonous consequence of increasing parameter $\alpha$ nicely demonstrated by the middle row where red cooperators dominate the system. The remaining parameters are set as $\beta = 0.3$, $\gamma = 0.95$, $b = 1.05$ and $\epsilon = 0.01$.
• Figure 5: Time evolution of microscopic transition probabilities at different values of $\alpha$. The three panels show the cases obtained at $\alpha=0$, 0.1 and 0.3. As shown in the legend, red line represents the likelihood that a cooperator players remains cooperator, and blue line the probability when a defector keeps its strategy. Pink line depicts the probability that a defector becomes cooperator, while green line marks the opposite process. Note that green is hardly seen, covered by pink, because strategy change processes are equally frequent in the stationary state. Interestingly, the less intensive individual movements can be observed in the middle panel, which results in the highest general cooperation level. All data are obtained under the parameter conditions of $\beta=0.3$, $\gamma=0.95$, $b=1.05$ and $\epsilon = 0.01$.