Demystifying Spectral Feature Learning for Instrumental Variable Regression

TL;DR

This work investigates spectral-feature learning for nonparametric instrumental variable regression (NPIV). It shows that the performance of sieve 2SLS with spectral features hinges on two factors: how well the structural function h0 aligns with the leading eigenfunctions of the conditional expectation operator T and how slowly T’s singular values decay. The authors derive a sharp generalization bound, connect the spectral-contrastive learning objective to a Hilbert-Schmidt approximation of Td, and prove a data-driven regime-diagnosis procedure that estimates spectral alignment and decay from data. They validate the theory with synthetic experiments revealing a clear good–bad–ugly taxonomy and apply the diagnosis to the dSprites dataset, demonstrating practical utility. Overall, the work provides both theoretical guarantees and actionable diagnostics for when spectral-feature NPIV methods will succeed in practice and how to adapt data collection accordingly.

Abstract

We address the problem of causal effect estimation in the presence of hidden confounders, using nonparametric instrumental variable (IV) regression. A leading strategy employs spectral features - that is, learned features spanning the top eigensubspaces of the operator linking treatments to instruments. We derive a generalization error bound for a two-stage least squares estimator based on spectral features, and gain insights into the method's performance and failure modes. We show that performance depends on two key factors, leading to a clear taxonomy of outcomes. In a good scenario, the approach is optimal. This occurs with strong spectral alignment, meaning the structural function is well-represented by the top eigenfunctions of the conditional operator, coupled with this operator's slow eigenvalue decay, indicating a strong instrument. Performance degrades in a bad scenario: spectral alignment remains strong, but rapid eigenvalue decay (indicating a weaker instrument) demands significantly more samples for effective feature learning. Finally, in the ugly scenario, weak spectral alignment causes the method to fail, regardless of the eigenvalues' characteristics. Our synthetic experiments empirically validate this taxonomy. We further introduce a practical procedure to estimate these spectral properties from data, allowing practitioners to diagnose which regime a given problem falls into. We apply this method to the dSprites dataset, demonstrating its utility.

Figures (7)

• Figure 1: Dependence of the MSE of instrumental variable regression on the decay rates of the spectrum and coefficients of the structural function. IV fitting and the MSE evaluations were repeated 500 times per parameter set, rendering the standard error of these estimates negligible. Left: Oracle spectral features; Right: Learned features.
• Figure 2: Comparison of the spectral methods' performance depending on the alignment of $h_0$ with the singular functions of $\mathcal{T}$. Blue: True $h_0$; Orange: 2SLS estimate.
• Figure 3: Example image from the dSprites dataset.
• Figure 4: Comparison of a density corresponding to a set of learned spectral features (left) and the true data generating density (right).
• Figure 5: Qualitative improvement in the 2SLS estimate of $h_0$ as $k$ increases. When $k = 1$, $h_0$ is orthogonal to the singular functions of $\mathcal{T}$; when $k = d - 1$, it is fully contained in their span.

Theorems & Definitions (27)

• Proposition 1
• Definition 1
• Theorem 1: Theorem B.1. chen2018optimal
• Proposition 2: Lemma 1 blundell2007semi
• Definition 2: Spectral features
• Corollary 1: Sieve 2SLS with spectral features
• Theorem 2: Eckart--Young--Mirsky Theorem
• Theorem 3
• Proposition 3
• Lemma 1