HessFormer: Hessians at Foundation Scale

TL;DR

HessFormer addresses the gap in accessible second-order information for foundation-scale transformers by enabling distributed Hessian-vector computations via stochastic Lanczos quadrature within the PyTorch/Huggingface ecosystem. The method preserves data, tensor, and pipeline parallelism and uses Pearlmutters trick to enable double backward passes, yielding empirical Hessian spectra for a 70B-parameter model on a single multi-GPU node. Key findings include the presence of large negative outliers and data-dependent spectral shapes, as well as evidence for near-sparsity and rank degeneracy phenomena driven by finite precision and orthogonality constraints. While offering a significant methodological advance, the work also highlights practical limitations (spectral-moment constraints, iteration-induced ghosts, and compute efficiency) and discusses broader implications for optimization, safety, and model compression at scale.

Abstract

Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the hundreds of billions of parameters, methods that rely on Hessian vector products, are still limited to run on a single GPU and thus cannot even work for models in the billion parameter range. We release a software package \textbf{HessFormer}, which integrates nicely with the well known Transformers package and allows for distributed hessian vector computation across a single node with multiple GPUs. Underpinning our implementation is a distributed stochastic lanczos quadrature algorithm, which we release for public consumption. Using this package we investigate the Hessian spectral density of the recent Deepseek $70$bn parameter model.

Figures (9)

• Figure 1: Comparison between Training a $1.5$Bn parameter Qwen distilled Deepseek model on wikitrain for $4$ epochs using auto from huggingface on multiple GPUs against using a single large enough GPU training on cuda.
• Figure 2: Stem comparison of using Auto (blue) vs. not using Auto (red) for Hessian of the DeepSeek-distilled Qwen models across three datasets. The rightmost subfigure is an example of how a large model looks on a common dataset.
• Figure 3: DeepSeek Qwen 1.5 Bn: spectral effects of data subsampling and orthogonalization on the wikitext dataset.
• Figure 4: Random Data with Various (n=num_samples, l=seq_length) Hessian with $30$ Lanczos HVP iterations.
• Figure 5: Real News Data (subsampled to $1\%$) with various amounts of further subsampling Hessia for Qwen $1.5$Bn parameter DeepSeek distilled