Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning and Consumption-Savings Behavior

Brandon Kaplowitz

TL;DR

This paper investigates how reinforcement learning, implemented with a neural-network approximation of the continuation value, can explain two observed consumption patterns during downturns: higher MPCs among previously low-asset unemployed households and persistent consumption scarring from past unemployment. By modeling households as learning agents in a Markov decision process, the author shows that value-function approximation errors that evolve with experience can generate both higher MPCs and lower consumption, even without borrowing constraints. The approach reproduces key empirical findings from Ganong et al. (2024) and Malmendier & Shen (2024) within a unified, model-free learning framework, challenging traditional ex-ante heterogeneity or purely belief-updating explanations. The work highlights adaptive learning as a plausible mechanism shaping consumption beyond current income and assets, with implications for how policy transfers affect spending under uncertainty. Limitations include the quarterly frequency vs biennial data and the absence of explicit belief dynamics, suggesting avenues for richer model-based RL or POMDP formulations in future research.

Abstract

This paper demonstrates how reinforcement learning can explain two puzzling empirical patterns in household consumption behavior during economic downturns. I develop a model where agents use Q-learning with neural network approximation to make consumption-savings decisions under income uncertainty, departing from standard rational expectations assumptions. The model replicates two key findings from recent literature: (1) unemployed households with previously low liquid assets exhibit substantially higher marginal propensities to consume (MPCs) out of stimulus transfers compared to high-asset households (0.50 vs 0.34), even when neither group faces borrowing constraints, consistent with Ganong et al. (2024); and (2) households with more past unemployment experiences maintain persistently lower consumption levels after controlling for current economic conditions, a "scarring" effect documented by Malmendier and Shen (2024). Unlike existing explanations based on belief updating about income risk or ex-ante heterogeneity, the reinforcement learning mechanism generates both higher MPCs and lower consumption levels simultaneously through value function approximation errors that evolve with experience. Simulation results closely match the empirical estimates, suggesting that adaptive learning through reinforcement learning provides a unifying framework for understanding how past experiences shape current consumption behavior beyond what current economic conditions would predict.

Reinforcement Learning and Consumption-Savings Behavior

TL;DR

This paper investigates how reinforcement learning, implemented with a neural-network approximation of the continuation value, can explain two observed consumption patterns during downturns: higher MPCs among previously low-asset unemployed households and persistent consumption scarring from past unemployment. By modeling households as learning agents in a Markov decision process, the author shows that value-function approximation errors that evolve with experience can generate both higher MPCs and lower consumption, even without borrowing constraints. The approach reproduces key empirical findings from Ganong et al. (2024) and Malmendier & Shen (2024) within a unified, model-free learning framework, challenging traditional ex-ante heterogeneity or purely belief-updating explanations. The work highlights adaptive learning as a plausible mechanism shaping consumption beyond current income and assets, with implications for how policy transfers affect spending under uncertainty. Limitations include the quarterly frequency vs biennial data and the absence of explicit belief dynamics, suggesting avenues for richer model-based RL or POMDP formulations in future research.

Abstract

This paper demonstrates how reinforcement learning can explain two puzzling empirical patterns in household consumption behavior during economic downturns. I develop a model where agents use Q-learning with neural network approximation to make consumption-savings decisions under income uncertainty, departing from standard rational expectations assumptions. The model replicates two key findings from recent literature: (1) unemployed households with previously low liquid assets exhibit substantially higher marginal propensities to consume (MPCs) out of stimulus transfers compared to high-asset households (0.50 vs 0.34), even when neither group faces borrowing constraints, consistent with Ganong et al. (2024); and (2) households with more past unemployment experiences maintain persistently lower consumption levels after controlling for current economic conditions, a "scarring" effect documented by Malmendier and Shen (2024). Unlike existing explanations based on belief updating about income risk or ex-ante heterogeneity, the reinforcement learning mechanism generates both higher MPCs and lower consumption levels simultaneously through value function approximation errors that evolve with experience. Simulation results closely match the empirical estimates, suggesting that adaptive learning through reinforcement learning provides a unifying framework for understanding how past experiences shape current consumption behavior beyond what current economic conditions would predict.
Paper Structure (44 sections, 18 equations, 10 figures, 4 tables)

This paper contains 44 sections, 18 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation and Goals
  3. Literature
  4. Learning and Consumption
  5. Reinforcement Learning
  6. Marginal Propensities to Consume
  7. Scarring
  8. Marginals versus Averages
  9. Model
  10. Notation
  11. Definitions and Relationships
  12. Income Process
  13. Decision Problems
  14. Rational Benchmark
  15. Simulated Agent
  16. ...and 29 more sections

Figures (10)

  • Figure 1: Consumption policies at $t = 5$ for the experiment with repeated employment with one agent receiving a series of unemployment realizations and the other receiving a series of employment realizations. The blue line is the agent with unemployment experiences, while the red line is the agent with employment experiences. The dotted gray line is the rational agent's policy. The figure on the left represents the agent's consumption policy as a function of their assets (x-axis) and income while employed $y_e$, while on the right it represents the agent's consumption policy as a function of their assets and income while unemployed $y_u$. Value functions use a polynomial fit.
  • Figure 2: MPCs for the experiment with repeated employment and unemployment realizations after $t =5$ periods, where one agent receives a series of unemployment realizations and the other receives a series of employment realizations. The blue line is the agent with unemployment experiences, while the red line is the agent with employment experiences. The dotted gray line is the rational benchmark's policy. The figure on the left represents the agent's marginal propensity to consume as a function of their assets (x-axis) and income while employed $y_e$, while on the left it represents the agent's consumption policy as a function of their assets and income while unemployed $y_u$. This uses the polynomial fit to the value function generating local oscillations from the polynomial.
  • Figure A1: Consumption as a function of assets and income after 50 periods of learning. No polynomial smoothing is used. The blue line is the reinforcement learner, the orange line is the rational benchmark. 95th percentiles are plotted.
  • Figure A2: MPC as a function of assets and income after 50 periods of learning. No polynomial smoothing is used. The blue line shows the reinforcement learner, the orange line shows the rational benchmark. 95th percentile intervals are plotted.
  • Figure A3: Consumption policies at $t = 0$ for the experiment with repeated employment with one agent receiving a series of unemployment realizations and the other receiving a series of employment realizations. The blue line is the agent with unemployment experiences, while the red line is the agent with employment experiences. The dotted gray line is the rational agent's policy. The figure on the left represents the agent's consumption policy as a function of their assets (x-axis) and income while employed $y_e$, while on the right it represents the agent's consumption policy as a function of their assets and income while unemployed $y_u$.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Definition 3.1