Empirical Equilibria in Agent-based Economic systems with Learning agents

TL;DR

The paper advances the intersection of AI and economics by embedding reinforcement learning within an empirical game-theoretic framework to study equilibria in a heterogeneous agent-based macroeconomic model. It constructs a four-type ABM (households, firms, central bank, government) within an OpenAI Gym–style environment and applies Policy Space Response Oracle (PSRO) to generate strategies that converge toward Nash equilibria of the empirical game, contrasting them with independent MARL. Empirical results show PSRO yields lower regret and more equilibrium-like behavior, particularly in labor allocation and policy stability, while achieving comparable rewards with fewer training episodes. By linking PSRO with EGTA in economic ABMs, the work demonstrates a practical path to analyzing policy outcomes and future equilibria in complex, data-informed economic settings.

Abstract

We present an agent-based simulator for economic systems with heterogeneous households, firms, central bank, and government agents. These agents interact to define production, consumption, and monetary flow. Each agent type has distinct objectives, such as households seeking utility from consumption and the central bank targeting inflation and production. We define this multi-agent economic system using an OpenAI Gym-style environment, enabling agents to optimize their objectives through reinforcement learning. Standard multi-agent reinforcement learning (MARL) schemes, like independent learning, enable agents to learn concurrently but do not address whether the resulting strategies are at equilibrium. This study integrates the Policy Space Response Oracle (PSRO) algorithm, which has shown superior performance over independent MARL in games with homogeneous agents, with economic agent-based modeling. We use PSRO to develop agent policies approximating Nash equilibria of the empirical economic game, thereby linking to economic equilibria. Our results demonstrate that PSRO strategies achieve lower regret values than independent MARL strategies in our economic system with four agent types. This work aims to bridge artificial intelligence, economics, and empirical game theory towards future research.

Figures (10)

• Figure 1: Discounted cumulative rewards during training for IMARL (left) and PSRO (right).
• Figure 2: Household labor hours (left) and consumption (right) with IMARL (top) and PSRO (bottom). With PSRO, less-skilled H2 works more and consumes more than H1.
• Figure 3: Price (left) and wage (right) of firms with IMARL (top) and PSRO (bottom). While F2 is cheaper than F1 in both schemes, the contrast is lower with PSRO.
• Figure 4: Interest rate set by the central bank (left) and inflation (right) with IMARL and PSRO. PSRO uses fewer rate options than IMARL, while achieving target inflation.
• Figure 5: Tax rate (left) and total tax collected (right) by the government with IMARL and PSRO. The government is able to collect similar tax amounts from households while setting lower tax rates with PSRO due to higher labor hours.