Model-Agnostic and Uncertainty-Aware Dimensionality Reduction in Supervised Learning

TL;DR

This paper tackles the problem of identifying the smallest dimensional representation that preserves predictive utility in supervised learning. It introduces Predictiveness-Induced Order Determination (POD), a model-agnostic framework that directly evaluates out-of-sample predictiveness via cross-fitted risk gaps and a sequential testing procedure, yielding uncertainty bounds and consistency guarantees. Theoretical results establish the asymptotic null distribution, underestimation bounds, and consistency, while numerical studies on factor regression, SDR, and a PenDigits dataset demonstrate accurate order estimation and improved predictive performance. POD unifies dimension reduction with predictive performance, offering a principled, uncertainty-aware approach that adapts to the chosen loss and downstream learner. The work has practical implications for prediction-centric pipelines by providing a robust, flexible tool to decide how many reduced dimensions are truly necessary.

Abstract

Dimension reduction is a fundamental tool for analyzing high-dimensional data in supervised learning. Traditional methods for estimating intrinsic order often prioritize model-specific structural assumptions over predictive utility. This paper introduces predictive order determination (POD), a model-agnostic framework that determines the minimal predictively sufficient dimension by directly evaluating out-of-sample predictiveness. POD quantifies uncertainty via error bounds for over- and underestimation and achieves consistency under mild conditions. By unifying dimension reduction with predictive performance, POD applies flexibly across diverse reduction tasks and supervised learners. Simulations and real-data analyses show that POD delivers accurate, uncertainty-aware order estimates, making it a versatile component for prediction-centric pipelines.

Figures (6)

• Figure 1: Empirical probabilities of correct estimation ($\widehat{d}=d^*_{\rm F}$), overestimation ($\widehat{d}>d^*_{\rm F}$), and underestimation ($\widehat{d}<d^*_{\rm F}$) across $n$ for factor regression. Red dashed horizontal lines in the second row mark the nominal significance levels $\alpha=0.01$ and $\alpha=0.05$.
• Figure 2: Empirical probabilities of correct estimation ($\widehat{d}=d^*_{\rm CMS}$), overestimation ($\widehat{d}>d^*_{\rm CMS}$), and underestimation ($\widehat{d}<d^*_{\rm CMS}$) across $n$ for SDR. Red dashed horizontal lines in the second row mark the nominal significance levels $\alpha=0.01$ and $\alpha=0.05$.
• Figure 3: PenDigits handwritten digits analysis. Panel A shows the eigenvalue scree plot of the DR kernel matrix. Panel B visualizes the DR representation with the first two and three DR directions. Panel C displays the test risks for different dimensions.
• Figure S.1: Roadmap for the Supplementary Material.
• Figure S.2: Empirical probabilities of correct estimation ($\widehat{d}=d^*_{\rm CMS}$), overestimation ($\widehat{d}>d^*_{\rm CMS}$), and underestimation ($\widehat{d}<d^*_{\rm CMS}$) across $n$ for SDR. Red dashed horizontal lines in the second row mark the nominal significance levels $\alpha=0.01$ and $\alpha=0.05$.

Theorems & Definitions (37)

• Example 1: Factor regression
• Example 2: Sufficient dimension reduction, SDR
• Example 3: Reduced-rank regression
• Proposition 1
• Remark 1
• Remark 2
• Theorem 1: Asymptotic null distribution
• Proposition 2
• Theorem 2: Underestimation bound
• Theorem 3: Consistency