Reinforcement Learning Dynamics of Network Vaccination and Hysteresis: A Double-Edged Sword for Addressing Vaccine Hesitancy

TL;DR

An experience-based learning (Q-learning) dynamics model of vaccination behavior in social networks, where agents choose whether or not to vaccinate given environmental feedbacks from their local neighborhood, focuses on how bounded rationality of individuals impacts decision-making of irrational agents in networks.

Abstract

Mass vaccination remains a long-lasting challenge for disease control and prevention with upticks in vaccine hesitancy worldwide. Here, we introduce an experience-based learning (Q-learning) dynamics model of vaccination behavior in social networks, where agents choose whether or not to vaccinate given environmental feedbacks from their local neighborhood. We focus on how bounded rationality of individuals impacts decision-making of irrational agents in networks. Additionally, we observe hysteresis behavior and bistability with respect to vaccination cost and the Q-learning hyperparameters such as discount rate. Our results offer insight into the complexities of Q-learning and particularly how foresightedness of individuals will help mitigate - or conversely deteriorate, therefore acting as a double-edged sword - collective action problems in important contexts like vaccination. We also find a diversification of uptake choices, with individuals evolving into complete opt-in vs. complete opt-out. Our results have real-world implications for targeting the persistence of vaccine hesitancy using an interdisciplinary computational social science approach integrating social networks, game theory, and learning dynamics.

Figures (8)

• Figure 1: Payoff structure of the game and reinforcement learning process. Cooperating (vaccinating) incurs a cost $r_v$, and defecting means the agent does not get vaccinated, in which case they pay a cost $r_i$ if they become infected. We then use Q-learning to allow players to update their strategies between seasonal disease spread.
• Figure 2: Evolved decision-making network. The network is instantiated on a $k$-regular graph with degree $k=4$. Here node colors indicate the probability of vaccinating under the state of no vaccinated neighbors $s_t = 0$. A few unvaccinated are intermixed with vaccinated individuals. Parameter values to obtain this network were: $\alpha = 1, \beta = 0.4, \gamma = 0.4, d = 0.95, T = 0.01, r = 0.8$.
• Figure 3: Temporal learning dynamics of vaccination and infection. Over time, agents successfully learn the optimal strategy of an overall vaccination level of 91.3%. This figure was obtained by averaging the timescales of 100 independent runs. Parameter values to obtain these results: $\alpha = 0.1, \beta = 0.4, \gamma = 0.1, d = 0.8, r = 0.1, T = 0.01.$
• Figure 4: Context-dependent diversification of vaccination strategies. As the state (number of vaccinated neighbors) increases, the height of the right mode (vaccinate) decreases, and the height of the left mode (do not vaccinate) increases. This reveals a double-edged sword impact of network clustering of vaccination. Parameter values to obtain these results: $\alpha = 1, \beta = 0.4, \gamma = 0.5, d = 0.95, r = 0.2, T = 0.01.$
• Figure 5: Responsiveness to the basic reproduction ratio $R_0$. As the basic reproduction ratio increases, so does the likelihood of infection for unvaccinated individuals. In response, agents vaccinate at higher rates. At lower temperatures this response is much faster, resulting in higher uptake than at high temperatures. Parameter values to obtain these results: $\alpha = 0.1, d = 0.8, r = 0.1$.