Model-free identification in ill-posed regression

TL;DR

This work develops a model-free framework to address identifiability in ill-posed regression with highly correlated features by separating information into a relevant subspace $\mathcal{B}_{y}$ and an irrelevant complement $\mathcal{B}_{y}^{\perp}$. It defines parsimonious representations $\bm{\theta}_{\mathcal{B}}=(\mathcal{B},\mathbf{U}_{\mathcal{B}},r_{\mathcal{B}},\bm{\beta}_{\mathcal{B}})$ and analyzes a broad class of population and sample linear dimensionality reduction algorithms $\mathcal{A}$ that are adaptive (depend only on the relevant subspace) and parsimonious (use limited degrees of freedom). The paper derives necessary and sufficient conditions for sharp population and finite-sample error bounds, showing that only adaptive, parsimonious algorithms can attain optimal rates in this ill-posed setting; it also connects these identifiability results to prediction risk under arbitrary dependence between features and response. Practically, the framework clarifies when sparse, unsupervised, or sufficient-reduction methods provide reliable interpretability and guides the design of interpretable algorithms in high-dimensional, overparameterized regimes. The results have implications for fields like genomics (e.g., GWAS), where interpretability and stable, low-DoF representations are crucial for clinical insights and risk prediction.

Abstract

The problem of parsimonious parameter identification in possibly high-dimensional linear regression with highly correlated features is addressed. This problem is formalized as the estimation of the best, in a certain sense, linear combinations of the features that are relevant to the response variable. Importantly, the dependence between the features and the response is allowed to be arbitrary. Necessary and sufficient conditions for such parsimonious identification -- referred to as statistical interpretability -- are established for a broad class of linear dimensionality reduction algorithms. Sharp bounds on their estimation errors, with high probability, are derived. To our knowledge, this is the first formal framework that enables the definition and assessment of the interpretability of a broad class of algorithms. The results are specifically applied to methods based on sparse regression, unsupervised projection and sufficient reduction. The implications of employing such methods for prediction problems are discussed in the context of the prolific literature on overparametrized methods in the regime of benign overfitting.

Figures (3)

• Figure 1: A toy simulation of our framework with $n=1000$ and $p=3$. A sample of i.i.d. observations (black) in the three-dimensional space; the features have full rank $r_{\mathbf{x}}=3$; the subspace of irrelevant features (light green) has dimension $r_{y^\bot}=1$ and is the first direction of variation; the subspace of relevant features (dark blue) has dimension $r_{y}=2$.
• Figure 2: A toy simulation of our framework with $n=1000$ and $p=3$. PLOTS: the observations (black) have full rank; the main direction of variation (light green) is irrelevant for the response; the second direction of variation (light red) determines the response; the third direction of variation (light blue) is a small noise. TOP: the dataset is aligned with the (x,y,z)-axis and the problem is sparse. BOTTOM: the dataset is rotated by $30^\circ$ around the z-axis and the problem is not sparse.
• Figure 3: A toy simulation of our framework with $n=1000$ and $p=3$ as in Figure \ref{['fig:visual_all_sparse_rotated']}. PLOTS: the oracle direction of the features (light red); the sparse direction of the features estimated by Forward Subset Selection (dark blue); the principal direction of the features estimated by Principal Component Regression (dark green); the covariance direction of the features estimated by Partial Least Squares (dark red). TABLES: the angles (in degrees) between the oracle direction of the features and the directions estimated by $\operatorname{FSS}$, $\operatorname{PCR}$ and $\operatorname{PLS}$ methods. LEFT: sparse setting where the dataset is aligned with the (x,y,z)-axis. RIGHT: non-sparse setting where the dataset is rotated by $30^\circ$ around the z-axis.

Theorems & Definitions (58)

• Lemma 2.2
• Theorem 2.4
• Corollary 2.5
• Remark 2.6: Our Assumptions
• Remark 2.7: Misleading Sparse Reduction
• Remark 2.8: Misleading Unsupervised Reduction
• Remark 2.9: Reliable Sufficient Reduction
• Theorem 2.12
• Corollary 2.13
• Theorem 2.14