Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics

TL;DR

<3-5 sentence high-level summary>

Abstract

We present a new approach to formulating and solving heterogeneous agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let agents learn equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes.

Figures (11)

• Figure 1: Aggregate saving function $S(r, z)$ in the Huggett economy
• Figure 2: Computational graph
• Figure 3: Simulation Results
• Figure 4: Solution comparison for the PE problem: SRL vs VFI
• Figure 5: Consumption Policy Function with Price Lags $p_{t-1}$ as State Variable

Theorems & Definitions (1)

• Definition 1: Sequential restricted perceptions equilibrium