Heterogeneous RBCs via deep multi-agent reinforcement learning

TL;DR

The paper addresses the challenge of incorporating realistic agent heterogeneity into macroeconomic models by marrying deep multi-agent reinforcement learning (MARL) with Real Business Cycle dynamics. The proposed MARL-BC framework enables a population of heterogeneous RL households to drive macro outcomes, reproducing textbook RBC with a single agent and Krusell–Smith mean-field results with many identical agents, while also supporting richer, endogenous heterogeneity beyond traditional GE models. Key contributions include a concrete MARL environment with fixed agent productivities, a learning architecture using parameter sharing across agents, and demonstrations across RBC, KS, and generalized heterogeneous settings, along with open-source code. The work advances a synthesis of ABMs and GE approaches, enabling endogenously learned policies and scalable heterogeneity, with potential applications to inequality, productivity shocks, and policy design in macroeconomics.

Abstract

Current macroeconomic models with agent heterogeneity can be broadly divided into two main groups. Heterogeneous-agent general equilibrium (GE) models, such as those based on Heterogeneous Agents New Keynesian (HANK) or Krusell-Smith (KS) approaches, rely on GE and 'rational expectations', somewhat unrealistic assumptions that make the models very computationally cumbersome, which in turn limits the amount of heterogeneity that can be modelled. In contrast, agent-based models (ABMs) can flexibly encompass a large number of arbitrarily heterogeneous agents, but typically require the specification of explicit behavioural rules, which can lead to a lengthy trial-and-error model-development process. To address these limitations, we introduce MARL-BC, a framework that integrates deep multi-agent reinforcement learning (MARL) with Real Business Cycle (RBC) models. We demonstrate that MARL-BC can: (1) recover textbook RBC results when using a single agent; (2) recover the results of the mean-field KS model using a large number of identical agents; and (3) effectively simulate rich heterogeneity among agents, a hard task for traditional GE approaches. Our framework can be thought of as an ABM if used with a variety of heterogeneous interacting agents, and can reproduce GE results in limit cases. As such, it is a step towards a synthesis of these often opposed modelling paradigms.

Figures (9)

• Figure 1: The MARL-BC framework and two limit cases. The left orange square contains a schematic illustration of the MARL-BC framework. A population of $n$ heterogeneous RL household agents, $i=1, \dots, n$, possesses distinct productivities of capital $\kappa_i$ and of labour $\lambda_i$, and provides capitals $k^i_t$ and labours $\ell^i_t$. These individual inputs aggregate into total capital $K_t$ and total labour $L_t$, which drive production $Y_t$ via a Cobb-Douglas function. In the markets, the individual interest rates $r^i_t$ and the wages $w^i_t$ are assumed to be proportional to the corresponding individual productivities. Wealth $a^i_t$ is given by past depreciated capital $(1-\delta)k^i_t$, earned wages $w^i_t \ell^i_t$, and earned interest $r^i_t k^i_t$. Each agent needs to decide the fraction $\hat{c}^i_t$ of wealth to consume and the amount of labour to provide. Consumption and labour go into the instantaneous reward $R^i_t$ of the agent, which the agent learns to maximise in the long run with a discount factor $\beta$. The two right orange boxes represent classic macroeconomic models as limiting cases of our framework. The RBC model (top) is obtained using a representative agent ($n = 1$) and an AR(1) process for the technology shocks $A_t$. The Krusell–Smith mean-field model (bottom) is obtained using many ex-ante identical agents and technology and labour switching between discrete values.
• Figure 2: State-of-the-art RL schemes can solve the household optimisation problem. Mean rewards obtained by different RL schemes as a function of the number of training steps. All schemes considered succeed in learning a policy that maximises cumulative rewards. SAC, DDPG and TD3 are seen to perform significantly better than PPO in terms of sample efficiency. However, PPO's greater computational efficiency (not shown in this graph) can make also this method a good candidate for the task.
• Figure 3: MARL-BC can reproduce representative‐agent results with a single agent. The left panel shows how the RL household recovers the optimal choices of consumption fraction $\hat{c}$ and labour supply $\ell$ as given in Eq. \ref{['eq:textbook_rbc_actions']} for a textbook, analytically solvable, RBC model with full capital depreciation ($\delta=1)$. The centre panel shows a similar convergence, but for a more complex model with a partial depreciation ($\delta=0.025$). This model cannot be analytically solved, and the optimal choices are obtained here using numerical tools common in economics. Finally, the right panel shows that MARL-BC is also capable of reproducing the dynamical behaviour of the standard RBC models as captured by an impulse response function. The plot shows the impulse response for consumption following a productivity shock of one standard deviation, with the dashed orange line indicating the standard RBC result and the full blue line indicating the equivalent MARL-BC behaviour.
• Figure 4: MARL-BC can reproduce mean‐field Krusell–Smith results with a large number of identical agents. The left panels are scatter plots of the aggregate capital at two consecutive times $K_{t}$ and $K_{t+1}$, at four training snapshots, with progressively more training steps (namely $2^7$, $2^{11}$, $2^{19}$ and $2^{21}$, which approximately amount to $10^2$, $2\cdot10^{3}$, $5\cdot10^{4}$, and $2\cdot10^{6}$). A characteristic result of the KS model is the linear time evolution of aggregate capital (as shown in the first figure of the original paper KrusellSmith1998), which is here recovered with a sufficient training steps. The centre panels depict the wealth distributions of the agents before and after training has converged with a sufficiently large number of training steps. While the Gini index for the untrained model is 0.08, it moves to 0.18 for the trained model distribution, a number which is in line with the original work. Finally, the right panels illustrate the curves of the marginal propensities to consume learned by the MARL-BC agents. Analogously to the KS agent policy, the consumption policy learned by the MARL-BC agents is almost completely flat when wealth is high, while it increases sharply when wealth is low.
• Figure 5: MARL-BC allows for modelling rich ex-ante heterogeneity and for the emergence of heterogeneous behaviour. The left panel reports the Lorenz curves, and the corresponding Gini indices, of the wealth distributions computed using the MARL-BC framework, either with the standard KS setting (blue dots) or with the extended KS with heterogeneous capital productivities (orange circles and green triangles). The graph illustrates how MARL-BC allows for the modelling of a much wider range of wealth inequalities. The centre panel reports the marginal propensities to consume of the three classes of agents (of low, medium and high returns) for the KS model with the Gini index equal to 0.61. The graph illustrates how MARL-BC gives rise to very different choices for the three household groups. The right panel illustrates the kind of wealth distribution that one can obtain by setting up an RBC model with 9 agents, each with a distinct combination of labour and capital productivity. Higher productivities also imply higher wages or capital returns that, in turn, give rise to higher capital accumulation.