Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Björn Roman Kohlberger

Abstract

The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.

Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

Abstract

The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.

Paper Structure

This paper contains 34 sections, 3 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Low-rank CNN training (ELRT).
  5. Riemannian fine-tuning (StelLA).
  6. Low-rank adapters (LoRA).
  7. Low-rank + sparse pre-training (LOST).
  8. SVD training for CNNs.
  9. Post-training SVD compression.
  10. Memory-efficient gradient methods (GaLore).
  11. Riemannian optimization.
  12. Methodology
  13. Spectral representation.
  14. Forward pass.
  15. Backward pass.
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Training memory at 70B scale. SCT requires 172$\times$ less memory than dense training.
  • Figure 2: Loss convergence for all ranks. All SCT configurations converge to the same loss floor ($\sim$4.2--4.5) regardless of rank. Dense converges to 1.29.
  • Figure 3: Left: Compression vs. quality Pareto frontier. Rank 128 achieves the best PPL at 11.7$\times$ compression. Right: GPU memory by method. Even rank 256 saves 40% of VRAM.