Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Rank Plus Sparse Matrix Transfer Learning under Growing Representations and Ambient Dimensions

Jinhang Chai, Xuyuan Liu, Elynn Chen, Yujun Yan

TL;DR

This work addresses transfer learning for structured matrix estimation when both ambient dimensions and latent representations grow across tasks. It introduces a subspace-anchored transfer framework that embeds a well-estimated source representation into a larger target space and estimates only low-rank innovations and sparse edits. A novel anchored alternating projection method provides deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding improved rates when increments are small. The approach is demonstrated on two canonical problems—Markov transition matrix estimation with dependent noise and structured covariance estimation—with both theoretical guarantees and empirical transfer gains, highlighting practical impact for continual representation expansion in large-scale learning systems.

Abstract

Learning systems often expand their ambient features or latent representations over time, embedding earlier representations into larger spaces with limited new latent structure. We study transfer learning for structured matrix estimation under simultaneous growth of the ambient dimension and the intrinsic representation, where a well-estimated source task is embedded as a subspace of a higher-dimensional target task. We propose a general transfer framework in which the target parameter decomposes into an embedded source component, low-dimensional low-rank innovations, and sparse edits, and develop an anchored alternating projection estimator that preserves transferred subspaces while estimating only low-dimensional innovations and sparse modifications. We establish deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding strictly improved rates when rank and sparsity increments are small. We demonstrate the generality of the framework by applying it to two canonical problems. For Markov transition matrix estimation from a single trajectory, we derive end-to-end theoretical guarantees under dependent noise. For structured covariance estimation under enlarged dimensions, we provide complementary theoretical analysis in the appendix and empirically validate consistent transfer gains.

Low-Rank Plus Sparse Matrix Transfer Learning under Growing Representations and Ambient Dimensions

TL;DR

This work addresses transfer learning for structured matrix estimation when both ambient dimensions and latent representations grow across tasks. It introduces a subspace-anchored transfer framework that embeds a well-estimated source representation into a larger target space and estimates only low-rank innovations and sparse edits. A novel anchored alternating projection method provides deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding improved rates when increments are small. The approach is demonstrated on two canonical problems—Markov transition matrix estimation with dependent noise and structured covariance estimation—with both theoretical guarantees and empirical transfer gains, highlighting practical impact for continual representation expansion in large-scale learning systems.

Abstract

Learning systems often expand their ambient features or latent representations over time, embedding earlier representations into larger spaces with limited new latent structure. We study transfer learning for structured matrix estimation under simultaneous growth of the ambient dimension and the intrinsic representation, where a well-estimated source task is embedded as a subspace of a higher-dimensional target task. We propose a general transfer framework in which the target parameter decomposes into an embedded source component, low-dimensional low-rank innovations, and sparse edits, and develop an anchored alternating projection estimator that preserves transferred subspaces while estimating only low-dimensional innovations and sparse modifications. We establish deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding strictly improved rates when rank and sparsity increments are small. We demonstrate the generality of the framework by applying it to two canonical problems. For Markov transition matrix estimation from a single trajectory, we derive end-to-end theoretical guarantees under dependent noise. For structured covariance estimation under enlarged dimensions, we provide complementary theoretical analysis in the appendix and empirically validate consistent transfer gains.
Paper Structure (25 sections, 3 theorems, 64 equations, 1 figure, 1 table, 3 algorithms)

This paper contains 25 sections, 3 theorems, 64 equations, 1 figure, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work and Our Distinction
  3. Problem Setup and Transfer Model
  4. General Transfer Setting ($M$ tasks)
  5. Ambient Feature Space Growth
  6. Representation Growth: Low-Rank and Sparse Transfer Structure
  7. Estimation Method: Anchored Low-Rank Plus Sparse Transfer
  8. Transfer Estimation Problem
  9. Anchored Low-Rank and Sparse Projections
  10. Transfer Alternating Projection Algorithm
  11. Computational Complexity and Practical Considerations
  12. Theoretical Guarantees (Non-Distributional)
  13. Application to Markov Transition Matrix Estimation
  14. Empirical Application to Structured Covariance Estimation
  15. Conclusion
  16. ...and 10 more sections

Key Result

Theorem 4.2

Let $(\widehat{L}^{(2)}, \widehat{S}^{(2)})$ be the output of Algorithm 1 applied to $Y^{(2)}$. Under Assumption asp:incoherence-sparsity, the following bound holds deterministically: where $\|W^{(2)}\|$ is the operator (spectral) norm of the noise matrix, and $\|W^{(2)}\|_{\max} = \max_{i,j} |W^{(2)}_{ij}|$ is the entrywise max norm.

Figures (1)

  • Figure 1: Comparison of estimation error under enlarged dimension for low-rank recovery and subspace estimation. We vary the target sample size ($n_2$) (with ($n_1$) fixed) and compare Transfer, Non-Transfer, and single-task PCA. Transfer consistently achieves lower error, with the largest gains in the low-sample regime.

Theorems & Definitions (8)

  • Remark 2.1: Identifiability
  • Theorem 4.2
  • Remark 4.3: Source estimation
  • Corollary 4.4
  • Corollary 5.1: Transfer Markov transition estimation
  • Remark 5.2: Comparison to non-transfer and regime interpretation
  • Remark 5.3: Connection to feature-based reinforcement learning
  • proof