Exploring a Principled Framework for Deep Subspace Clustering

TL;DR

This work addresses the weakness of existing deep subspace clustering methods, which often suffer from feature collapse and lack theoretical guarantees to recover a union-of-subspaces structure. It introduces PRO-DSC, a principled framework that regularizes the learned representations with a log-determinant term, enabling simultaneous learning of structured representations and self-expressive coefficients. The authors prove eigenspace alignment between the representation Gram matrix and the self-expressive error, derive conditions to prevent collapse, and show that, under certain regimes, learned representations form a union of orthogonal subspaces with a block-diagonal self-expressive matrix. A scalable implementation via reparameterization, Sinkhorn-based coefficient learning, and differential programming supports large datasets and unseen samples. Extensive experiments on synthetic data and six real-world benchmarks, using CLIP and BYOL pre-trained features, demonstrate superior clustering performance and qualitative evidence of noncollapse and UoS structure, with code provided for reproducibility.

Abstract

Subspace clustering is a classical unsupervised learning task, built on a basic assumption that high-dimensional data can be approximated by a union of subspaces (UoS). Nevertheless, the real-world data are often deviating from the UoS assumption. To address this challenge, state-of-the-art deep subspace clustering algorithms attempt to jointly learn UoS representations and self-expressive coefficients. However, the general framework of the existing algorithms suffers from a catastrophic feature collapse and lacks a theoretical guarantee to learn desired UoS representation. In this paper, we present a Principled fRamewOrk for Deep Subspace Clustering (PRO-DSC), which is designed to learn structured representations and self-expressive coefficients in a unified manner. Specifically, in PRO-DSC, we incorporate an effective regularization on the learned representations into the self-expressive model, prove that the regularized self-expressive model is able to prevent feature space collapse, and demonstrate that the learned optimal representations under certain condition lie on a union of orthogonal subspaces. Moreover, we provide a scalable and efficient approach to implement our PRO-DSC and conduct extensive experiments to verify our theoretical findings and demonstrate the superior performance of our proposed deep subspace clustering approach. The code is available at https://github.com/mengxianghan123/PRO-DSC.

Figures (15)

• Figure 1: Empirical Validation to Eigenspace Alignment and Noncollapse Representation in Mini-batch on CIFAR-100. (a): Alignment error curve during the training period. (b): Eigenspace correlation curves measured via $\langle{\bm{u}}_j,\frac{{\bm{G}}_b{\bm{u}}_j}{\|{\bm{G}}_b{\bm{u}}_j\|_2}\rangle$ for $j=1,\cdots,n_b$. (c) and (d): Eigenvalue curves.
• Figure 2: Empirical Validation to Noncollapse Representation on CIFAR-10 and CIFAR-100. Clustering accuracy (ACC%) and subspace-preserving representation error (SRE%) are displayed under varying $\alpha$ and $\gamma$. When collapse occurs, both ACC and SRE dramatically degenerate. The perceivable phase transition phenomenon is consistent with the condition to avoid collapse.
• Figure 3: Empirical Validation to Structured Representation on CIFAR-10. Gram matrices for CLIP features ${\bm{X}}$ and learned representations ${\bm{Z}}$ are shown in (a) and (b); whereas Data visualization of the samples from three categories ${\bm{X}}_{(3)}$ and ${\bm{Z}}_{(3)}$ via PCA are shown in (c) and (d), respectively.
• Figure 4: Empirical validation to Theorem \ref{['Theorem:orthogonal']} in Mini-batch on CIFAR-10. The mean curves of the absolute values of the in-block-diagonal entries (thick) and the off-block-diagonal entries (thin) are displayed along with the CSC condition (gray) during training PRO-DSC.
• Figure 5: Visualization Experiments on Synthetic Data.

Theorems & Definitions (13)

• Lemma 1: Haeffele:ICLR21
• Theorem 1: Eigenspace Alignment
• Theorem 2: Noncollapse Representation
• Theorem 3
• Lemma 1: Haeffele:ICLR21
• proof
• Lemma A1
• Theorem 1
• proof
• Theorem 2