Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering
Maksym Shamrai
TL;DR
This paper tackles the problem of grouping matrices for joint low-rank representation via concatenated SVD under explicit reconstruction budgets. It derives two spectral bounds—the Weyl-based monotone bound and a residual-based bound—for the SVD error of horizontally concatenated blocks, enabling global error control within clusters; an incremental truncated SVD estimator further enables scalable, approximate error tracking without forming the full concatenated matrix. Based on these results, three compression-aware clustering algorithms are proposed, each merging blocks only when the predicted joint SVD error remains below the user-specified budget \(\varepsilon\); the approach provides explicit guarantees (where possible) and practical performance across diverse datasets. Empirical evaluation demonstrates meaningful compression gains while maintaining error budgets, and contrasts with classical clustering baselines that fail to enforce spectral reconstruction constraints. The framework offers a principled alternative to heuristic block-grouping in concatenated SVD pipelines, with potential extensions to online, task-aware, and rank-adaptive clustering.
Abstract
Large collections of matrices arise throughout modern machine learning, signal processing, and scientific computing, where they are commonly compressed by concatenation followed by truncated singular value decomposition (SVD). This strategy enables parameter sharing and efficient reconstruction and has been widely adopted across domains ranging from multi-view learning and signal processing to neural network compression. However, it leaves a fundamental question unanswered: which matrices can be safely concatenated and compressed together under explicit reconstruction error constraints? Existing approaches rely on heuristic or architecture-specific grouping and provide no principled guarantees on the resulting SVD approximation error. In the present work, we introduce a theory-driven framework for compression-aware clustering of matrices under SVD compression constraints. Our analysis establishes new spectral bounds for horizontally concatenated matrices, deriving global upper bounds on the optimal rank-$r$ SVD reconstruction error from lower bounds on singular value growth. The first bound follows from Weyl-type monotonicity under blockwise extensions, while the second leverages singular values of incremental residuals to yield tighter, per-block guarantees. We further develop an efficient approximate estimator based on incremental truncated SVD that tracks dominant singular values without forming the full concatenated matrix. Therefore, we propose three clustering algorithms that merge matrices only when their predicted joint SVD compression error remains below a user-specified threshold. The algorithms span a trade-off between speed, provable accuracy, and scalability, enabling compression-aware clustering with explicit error control. Code is available online.