The Hessian of tall-skinny networks is easy to invert
Ali Rahimi
TL;DR
This work tackles the problem of efficiently applying the inverse Hessian to a vector for deep neural networks without forming or storing the full Hessian. It develops a matrix-based reexpression of backpropagation that lifts the Hessian into a block-tri-diagonal system, enabling a Hessian-inverse-vector product with complexity $O\left(L \max(a,p)^3\right)$ and linear scaling in the number of layers $L$. The approach leverages a tall-skinny network structure and a pivoted LDU factorization to solve a sparse block system, while drawing connections to Pearlmutter's Hessian-vector product and potential preconditioning use. The method aims to rekindle interest in deeper architectures by offering a practical path to second-order information without prohibitive memory or compute costs.
Abstract
We describe an exact algorithm for solving linear systems $Hx=b$ where $H$ is the Hessian of a deep net. The method computes Hessian-inverse-vector products without storing the Hessian or its inverse in time and storage that scale linearly in the number of layers. Compared to the naive approach of first computing the Hessian, then solving the linear system, which takes storage that's quadratic in the number of parameters and cubically many operations, our Hessian-inverse-vector product method scales roughly like Pearlmutter's algorithm for computing Hessian-vector products.