Policy-focused Agent-based Modeling using RL Behavioral Models

TL;DR

The paper argues that RL-based behavioral models provide a flexible, utility-maximizing framework for ABMs, potentially improving validity and policy relevance. It develops single- and multi-agent RL approaches, including a Multi-agent Actor-Critic (MAC) mechanism, and tests them on a Minority Game ABM and a Flu Transmission ABM. Results show RL agents can learn reward-seeking behaviors and sometimes outperform default heuristics, though generalization and memory requirements pose challenges. Overall, RL offers a scalable, adaptable blueprint for incorporating adaptive, behaviorally-valid decision-making into policy-focused ABMs.

Abstract

Agent-based Models (ABMs) are valuable tools for policy analysis. ABMs help analysts explore the emergent consequences of policy interventions in multi-agent decision-making settings. But the validity of inferences drawn from ABM explorations depends on the quality of the ABM agents' behavioral models. Standard specifications of agent behavioral models rely either on heuristic decision-making rules or on regressions trained on past data. Both prior specification modes have limitations. This paper examines the value of reinforcement learning (RL) models as adaptive, high-performing, and behaviorally-valid models of agent decision-making in ABMs. We test the hypothesis that RL agents are effective as utility-maximizing agents in policy ABMs. We also address the problem of adapting RL algorithms to handle multi-agency in games by adapting and extending methods from recent literature. We evaluate the performance of such RL-based ABM agents via experiments on two policy-relevant ABMs: a minority game ABM, and an ABM of Influenza Transmission. We run some analytic experiments on our AI-equipped ABMs e.g. explorations of the effects of behavioral heterogeneity in a population and the emergence of synchronization in a population. The experiments show that RL behavioral models are effective at producing reward-seeking or reward-maximizing behaviors in ABM agents. Furthermore, RL behavioral models can learn to outperform the default adaptive behavioral models in the two ABMs examined.

Figures (9)

• Figure 1: : Schematic of interaction components between an agent and the ABM environment (arrows extending out of boxes denote exposed signals).
• Figure 2: Time series of attendees for 500 time steps, 301 agents each with 2 strategies and a memory of 2
• Figure 3: A trained RL learner playing against 100 different randomly drawn populations
• Figure 4: Two different rounds of training with a neural network with 3 time steps of memory. In each round, there were 400 training steps with an episode length of 500 time steps. The black line is a rolling mean with a window length of 20.
• Figure 5: Training with an RL agent with 5 time steps of memory. In each round, there were 400 training steps with an episode length of 500 time steps. The low spikes are the case where the initial distribution of agents yields ties.