Directly Constructing Low-Dimensional Solution Subspaces in Deep Neural Networks

TL;DR

The paper addresses whether high ambient widths are necessary for state-of-the-art performance by positing that the true solution resides in a low-dimensional subspace $\tilde h \in \mathbb{R}^k$ obtained from $h \in \mathbb{R}^d$ via a fixed projection $\tilde h = \frac{1}{\sqrt{k}} R h$. Using Johnson-Lindenstrauss projections, the authors freeze backbones (ResNet-50, ViT, BERT) and train a linear head on $\tilde h$, achieving near-baseline accuracy under compression up to $16\times$, across CIFAR-100, ImageNet-100, and MNLI. This empirical robustness supports a constructive view of the solution manifold and motivates Subspace-Native Distillation, where future students would directly target the low-dimensional target instead of navigating the high-dimensional ambient space. The approach promises practical impact by enabling efficient deployment of compact models without retraining large backbones, while offering a new theoretical lens on the geometry of deep representations and optimization landscapes.

Abstract

While it is well-established that the weight matrices and feature manifolds of deep neural networks exhibit a low Intrinsic Dimension (ID), current state-of-the-art models still rely on massive high-dimensional widths. This redundancy is not required for representation, but is strictly necessary to solve the non-convex optimization search problem-finding a global minimum, which remains intractable for compact networks. In this work, we propose a constructive approach to bypass this optimization bottleneck. By decoupling the solution geometry from the ambient search space, we empirically demonstrate across ResNet-50, ViT, and BERT that the classification head can be compressed by even huge factors of 16 with negligible performance degradation. This motivates Subspace-Native Distillation as a novel paradigm: by defining the target directly in this constructed subspace, we provide a stable geometric coordinate system for student models, potentially allowing them to circumvent the high-dimensional search problem entirely and realize the vision of Train Big, Deploy Small.