Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subspace Ordering for Maximum Response Preservation in Sufficient Dimension Reduction

Derik T. Boonstra, Rakheon Kim, Dean M. Young

TL;DR

This work challenges the conventional eigenvalue-based ordering of SDR directions and introduces predictive-importance criteria that directly measure a subspace's relevance to the response. By defining a binary-response T-statistic based measure $T_j$ and a multiclass/continuous-extension via the F-statistic $F_j$ (and population analogues $\Delta_j$, $\Psi_j$), the authors establish consistency results and demonstrate through simulations and real data that ordering by these criteria yields improved prediction and subspace recovery. The approach unifies discriminant analysis perspectives with SDR and shows that traditional variance-focused ordering can miss the most informative directions for prediction. The proposed criteria can turn unsupervised methods into pseudo-supervised ones and are supported by publicly available code, offering a practical, interpretable alternative for subspace selection in SDR tasks.

Abstract

Sufficient dimension reduction (SDR) methods aim to identify a dimension reduction subspace (DRS) that preserves all the information about the conditional distribution of a response given its predictor. Traditional SDR methods determine the DRS by solving a method-specific generalized eigenvalue problem and selecting the eigenvectors corresponding to the largest eigenvalues. In this article, we argue against the long-standing convention of using eigenvalues as the measure of subspace importance and propose alternative ordering criteria that directly assess the predictive relevance of each subspace. For a binary response, we introduce a subspace ordering criterion based on the absolute value of the independent Student's T-statistic. Theoretically, our criterion identifies subspaces that achieve the local minimum Bayes' error rate and yields consistent ordering of directions under mild regularity conditions. Additionally, we employ an F-statistic to provide a framework that unifies categorical and continuous responses under a single subspace criterion. We evaluate our proposed criteria within multiple SDR methods through extensive simulation studies and applications to real data. Our empirical results demonstrate the efficacy of reordering subspaces using our proposed criteria, which generally improves classification accuracy and subspace estimation compared to ordering by eigenvalues.

Subspace Ordering for Maximum Response Preservation in Sufficient Dimension Reduction

TL;DR

This work challenges the conventional eigenvalue-based ordering of SDR directions and introduces predictive-importance criteria that directly measure a subspace's relevance to the response. By defining a binary-response T-statistic based measure and a multiclass/continuous-extension via the F-statistic (and population analogues , ), the authors establish consistency results and demonstrate through simulations and real data that ordering by these criteria yields improved prediction and subspace recovery. The approach unifies discriminant analysis perspectives with SDR and shows that traditional variance-focused ordering can miss the most informative directions for prediction. The proposed criteria can turn unsupervised methods into pseudo-supervised ones and are supported by publicly available code, offering a practical, interpretable alternative for subspace selection in SDR tasks.

Abstract

Sufficient dimension reduction (SDR) methods aim to identify a dimension reduction subspace (DRS) that preserves all the information about the conditional distribution of a response given its predictor. Traditional SDR methods determine the DRS by solving a method-specific generalized eigenvalue problem and selecting the eigenvectors corresponding to the largest eigenvalues. In this article, we argue against the long-standing convention of using eigenvalues as the measure of subspace importance and propose alternative ordering criteria that directly assess the predictive relevance of each subspace. For a binary response, we introduce a subspace ordering criterion based on the absolute value of the independent Student's T-statistic. Theoretically, our criterion identifies subspaces that achieve the local minimum Bayes' error rate and yields consistent ordering of directions under mild regularity conditions. Additionally, we employ an F-statistic to provide a framework that unifies categorical and continuous responses under a single subspace criterion. We evaluate our proposed criteria within multiple SDR methods through extensive simulation studies and applications to real data. Our empirical results demonstrate the efficacy of reordering subspaces using our proposed criteria, which generally improves classification accuracy and subspace estimation compared to ordering by eigenvalues.

Paper Structure

This paper contains 16 sections, 10 theorems, 18 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. A Review of Methodologies
  3. Notation
  4. Discriminant Analysis
  5. Select SDR Methods
  6. Subspace Ordering Criterion for Binary Response Preservation
  7. Method
  8. Theoretical Properties
  9. Subspace Criterion for a Categorical or Continuous Response
  10. Simulation Studies
  11. Simulation for Binary Response
  12. Simulation for Continuous Response
  13. Real Data Applications
  14. Application to High-Dimensional Gene Expression Data
  15. Application to Real Data with a Continuous Response
  16. ...and 1 more sections

Key Result

Lemma 1

Let $\mathcal{L}$ and $\mathcal{Q}$ be as defined in eq:qda_L and eq:qda_Q, respectively. Then, under model eq:model, $\mathcal{S}_{Y \mid \mathbf{X}} = \mathcal{S}_{D(Y \mid \mathbf{X})} = \mathcal{L} \cup \mathcal{Q}$.

Figures (6)

  • Figure 1: Simulated data from two multivariate normal populations described in Section \ref{['subsec:ill_ex']} in the subspaces spanned by the leading eigenvectors $\mathbf{v}_1$, $\mathbf{v}_{2}$, and $\mathbf{v}_{3}$. Vertical jitter is added for visualization purposes.
  • Figure 2: Estimated conditional error rate $(\widehat{CER})$ plots for the simulation described in Section \ref{['subsec:5.1_CER_sims']} for contrasting eigenvalue-ordered SDR methods (labeled by their standard names) with their $T_{j}$-ordered counterparts (denoted by the superscript T). All SDR methods used the QDA classifier, and the horizontal line represents the median $\widehat{CER}$ using QDA without dimension reduction.
  • Figure 3: Estimated conditional error rate $(\widehat{CER})$ plots for the simulation described in Section \ref{['subsec:5.1_CER_sims']}, contrasting eigenvalue-ordered SDR methods (labeled by their standard names) with their $T_{j}$-ordered counterparts (denoted by the superscript $T$). LDA was used as the supervised classifier for all SDR methods. The median $\widehat{CER}$ using LDA without dimension reduction is represented by the horizontal line.
  • Figure 4: Estimated subspace distance $\mathcal{D}$, given in \ref{['eq:D']}, plots for contrasting eigenvalue-ordered SDR methods (labeled by their standard names) with their $F_{j}$-ordered counterparts (denoted by the superscript $F$). The number of slices was set to $H = 5$ for all SDR methods.
  • Figure 5: Leukemia data discussed in Section \ref{['subsec:6.1_golub']} in the estimated SIR-II subspaces corresponding to the largest eigenvalue (left plot) and the largest $T_{j}$ value (right plot). QDA was used for the estimated decision boundaries. Vertical jitter was added for visualization.
  • ...and 1 more figures

Theorems & Definitions (10)

  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Theorem 1
  • Theorem 2
  • Lemma 4
  • Theorem 3
  • Corollary 1
  • Theorem 4
  • Theorem 5