Machine learning the first stage in 2SLS: Practical guidance from bias decomposition and simulation

TL;DR

The paper tackles the practical question of using machine learning in the first stage of 2SLS for causal inference. It develops a three-component bias decomposition and uses two data-generating processes (a low-complexity case with 7 strong instruments and a high-complexity case with 100 mixed instruments) to compare traditional 2SLS, ML-curated instrument sets (post-Lasso, PCA), and ML-first-stage methods (Lasso, NN, RF, boosting). The main finding is that while linear ML approaches can match or improve performance in some settings, most nonlinear ML methods substantially increase second-stage bias, sometimes beyond the bias of endogenous OLS; a careful bias decomposition reveals why (correlation between first-stage predictions and residuals, overfitting endogenous variation, and variance reduction amplifying bias). The paper discusses MLSS-based remedies and concludes that naive ML in the first stage offers limited gains and can incur substantial costs, motivating cautious use and targeted instrument-crafting approaches within 2SLS. Overall, the work provides actionable guidance on when ML can aid or harm 2SLS and highlights the need for stronger exogeneity assumptions or alternative ML-integrated IV methods.

Abstract

Machine learning (ML) primarily evolved to solve "prediction problems." The first stage of two-stage least squares (2SLS) is a prediction problem, suggesting potential gains from ML first-stage assistance. However, little guidance exists on when ML helps 2SLS$\unicode{x2014}$or when it hurts. We investigate the implications of inserting ML into 2SLS, decomposing the bias into three informative components. Mechanically, ML-in-2SLS procedures face issues common to prediction and causal-inference settings$\unicode{x2014}$and their interaction. Through simulation, we show linear ML methods (e.g., post-Lasso) work well, while nonlinear methods (e.g., random forests, neural nets) generate substantial bias in second-stage estimates$\unicode{x2014}$potentially exceeding the bias of endogenous OLS.

Figures (9)

• Figure 1: The "high-complexity" DGP: 100 instruments of varying strength
• Figure 2: Main results---$\hat{\beta}$ distributions across competing two-stage methods
• Figure 3: Exclusion-restriction violations via higher-order interactions among instruments ('low-complexity case' of 7 strong instruments)
• Figure A1: Predictions vs. estimation: Comparing cross-validated prediction performance with bias in random-forest-based 2SLS
• Figure A2: Distributions of estimates with "exclusion-restriction violations" from $k$-term interactions ('low-complexity case)