Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mode-wise Principal Subspace Pursuit and Matrix Spiked Covariance Model

Runshi Tang, Ming Yuan, Anru R. Zhang

TL;DR

This work introduces Mode-wise Principal Subspace Pursuit (MOP-UP) to perform joint row- and column-wise dimensionality reduction on collections of matrices by leveraging matrix-variate spiked covariance models and their higher-order generalizations. The method combines Average Subspace Capture (ASC) for initialization with Alternating Projection (AP) to iteratively refine shared row and column subspaces, underpinned by a novel blockwise eigenspace perturbation bound that tightens perturbation analysis beyond classical results. Theoretical contributions include finite-sample error bounds for initialization and linear convergence of AP, plus a global high-probability theory for MOP-UP and its tensor generalizations. Empirically, MOP-UP demonstrates effective dimension reduction and improved downstream performance on MNIST and fMRI datasets, while offering a clear path to higher-order tensor extensions with identifiability guarantees. These results provide a versatile, scalable framework for covariance-aware dimension reduction in multi-subject matrix and tensor data.

Abstract

This paper introduces a novel framework called Mode-wise Principal Subspace Pursuit (MOP-UP) to extract hidden variations in both the row and column dimensions for matrix data. To enhance the understanding of the framework, we introduce a class of matrix-variate spiked covariance models that serve as inspiration for the development of the MOP-UP algorithm. The MOP-UP algorithm consists of two steps: Average Subspace Capture (ASC) and Alternating Projection (AP). These steps are specifically designed to capture the row-wise and column-wise dimension-reduced subspaces which contain the most informative features of the data. ASC utilizes a novel average projection operator as initialization and achieves exact recovery in the noiseless setting. We analyze the convergence and non-asymptotic error bounds of MOP-UP, introducing a blockwise matrix eigenvalue perturbation bound that proves the desired bound, where classic perturbation bounds fail. The effectiveness and practical merits of the proposed framework are demonstrated through experiments on both simulated and real datasets. Lastly, we discuss generalizations of our approach to higher-order data.

Mode-wise Principal Subspace Pursuit and Matrix Spiked Covariance Model

TL;DR

This work introduces Mode-wise Principal Subspace Pursuit (MOP-UP) to perform joint row- and column-wise dimensionality reduction on collections of matrices by leveraging matrix-variate spiked covariance models and their higher-order generalizations. The method combines Average Subspace Capture (ASC) for initialization with Alternating Projection (AP) to iteratively refine shared row and column subspaces, underpinned by a novel blockwise eigenspace perturbation bound that tightens perturbation analysis beyond classical results. Theoretical contributions include finite-sample error bounds for initialization and linear convergence of AP, plus a global high-probability theory for MOP-UP and its tensor generalizations. Empirically, MOP-UP demonstrates effective dimension reduction and improved downstream performance on MNIST and fMRI datasets, while offering a clear path to higher-order tensor extensions with identifiability guarantees. These results provide a versatile, scalable framework for covariance-aware dimension reduction in multi-subject matrix and tensor data.

Abstract

This paper introduces a novel framework called Mode-wise Principal Subspace Pursuit (MOP-UP) to extract hidden variations in both the row and column dimensions for matrix data. To enhance the understanding of the framework, we introduce a class of matrix-variate spiked covariance models that serve as inspiration for the development of the MOP-UP algorithm. The MOP-UP algorithm consists of two steps: Average Subspace Capture (ASC) and Alternating Projection (AP). These steps are specifically designed to capture the row-wise and column-wise dimension-reduced subspaces which contain the most informative features of the data. ASC utilizes a novel average projection operator as initialization and achieves exact recovery in the noiseless setting. We analyze the convergence and non-asymptotic error bounds of MOP-UP, introducing a blockwise matrix eigenvalue perturbation bound that proves the desired bound, where classic perturbation bounds fail. The effectiveness and practical merits of the proposed framework are demonstrated through experiments on both simulated and real datasets. Lastly, we discuss generalizations of our approach to higher-order data.
Paper Structure (40 sections, 49 theorems, 339 equations, 8 figures, 5 tables, 3 algorithms)

This paper contains 40 sections, 49 theorems, 339 equations, 8 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Matrix Spiked Covariance Models and Higher-order Generalizations
  3. Our Contributions
  4. Literature Review
  5. Organization
  6. Models
  7. Notation and Preliminaries
  8. Matrix Spiked Covariance Model
  9. Algorithm: MOP-UP
  10. Algorithm
  11. Interpretations
  12. Matrix Spiked Covariance Model versus Existing Models
  13. Theoretical Analysis
  14. Error Bound for Initialization via ASC
  15. Local Convergence of Iterations of AP
  16. ...and 25 more sections

Key Result

Theorem 1

$X\in \mathbb{R}^{p_1\times p_2}$ satisfies the high-order spiked covariance model if and only if there exist a deterministic matrix $M$, random matrices $B\in \mathbb{R}^{p_1\times r_2}$ and $A\in \mathbb{R}^{r_1\times p_2}$ with mean 0 such that Here, $U\in \mathbb{O}_{p_1, r_1}, V\in \mathbb{O}_{p_2, r_2}$ are fixed semi-orthogonal matrices, $Z\in \mathbb{R}^{p_1\times p_2}$ is a random matrix

Figures (8)

  • Figure 1: Illustration of a matrix spiked covariance model in a decomposition form
  • Figure 2: Comparison of accuracy: Mean accuracy across 10 folds versus rank $r = r_1 = r_2$ used as a hyperparameter in MPCA, 2D-LDA, and our proposed MOP-UP. The length of the error bar represents the standard deviation.
  • Figure 3: Visualization of dimension reduced digit "9" images by MOP-UP with $r = 3$
  • Figure 4: Visualization of dimension reduced digit "9" images by MPCA. The dimension-reduced digit "9" images by MPCA with $r = 3$ or $6$ ($r=6$ matches the top-left image of Figure \ref{['mnist_example']}) is unidentifiable.
  • Figure 5: Estimation error of MOP-UP with the varying value of $p_1$
  • ...and 3 more figures

Theorems & Definitions (88)

  • Definition 1: High-order Spiked Covariance Model Matrix Variate Case
  • Theorem 1: Equivalent Definitions for High-order Spiked Covariance
  • Theorem 2: Identifiability Condition for Matrix Spiked Covariance Model
  • Remark 1
  • Example 1: An Identifiable Example of Matrix Spiked Covariance Model
  • Example 2: An Unidentifiable Example of Matrix Spiked Covariance Model
  • Theorem 3
  • Corollary 1
  • Proposition 1
  • Theorem 4: Error bound of ASC in the noisy case
  • ...and 78 more