Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learning
Bo Dai, Na Li, Dale Schuurmans
TL;DR
The paper addresses the lack of a unified theory for SSL representations by introducing a spectral-sufficiency framework that expresses $P(y|x)$ through a pair of spectral components, enabling downstream prediction to be captured linearly as $\mathbb{E}[y|x] = \langle \varphi(x), w\rangle$ under unlabeled data. It unifies existing SSL methods via energy-based, latent-variable, and nonlinear spectral representations, and shows how classical spectral decompositions (e.g., mutual information via $D_{KL}$, NCE, and power iterations) underlie many SSL objectives. It further extends the framework to multi-modal learning, diffusion-inspired methods, and downstream tasks such as regression, causal inference, and reinforcement learning, illustrating broad applicability and potential for principled algorithm design. The practical upshot is a principled lens to analyze, compare, and extend SSL methods, guiding architectures and objectives toward representations that are both sufficient for a range of tasks and efficient to learn. The work also highlights tradeoffs between linear vs. nonlinear representations, and energy-based vs. latent-variable formulations, offering a roadmap for designing scalable, theoretically grounded SSL systems with broad impact across domains.
Abstract
Self-supervised learning (SSL) have improved empirical performance by unleashing the power of unlabeled data for practical applications. Specifically, SSL extracts the representation from massive unlabeled data, which will be transferred to a plenty of down streaming tasks with limited data. The significant improvement on diverse applications of representation learning has attracted increasing attention, resulting in a variety of dramatically different self-supervised learning objectives for representation extraction, with an assortment of learning procedures, but the lack of a clear and unified understanding. Such an absence hampers the ongoing development of representation learning, leaving a theoretical understanding missing, principles for efficient algorithm design unclear, and the use of representation learning methods in practice unjustified. The urgency for a unified framework is further motivated by the rapid growth in representation learning methods. In this paper, we are therefore compelled to develop a principled foundation of representation learning. We first theoretically investigate the sufficiency of the representation from a spectral representation view, which reveals the spectral essence of the existing successful SSL algorithms and paves the path to a unified framework for understanding and analysis. Such a framework work also inspires the development of more efficient and easy-to-use representation learning algorithms with principled way in real-world applications.