Unit-Consistent (UC) Adjoint for GSD and Backprop in Deep Learning Applications
Jeffrey Uhlmann
TL;DR
The paper addresses the problem that gradient-based learning in networks with positively homogeneous nonlinearities is sensitive to diagonal rescalings of internal coordinates, leading to gauge-dependent optimization. It introduces a unit-consistent (UC) adjoint by performing backpropagation in canonical coordinates defined via a diagonal decomposition $W = D W' E$, yielding the UC adjoint $W^{*_{\mathrm{UC}}} = E^{-1} W^{\top} D^{-1} = (W')^{\top}$ and a diagonal-scale invariant steepest-descent update $W^+ = W - \eta D^{2} G E^{2}$. The framework extends to biases, convolutions, residual connections, and stateful optimizers, proving equivariance and deriving explicit updates in original coordinates; it further argues for normalization by geometry over forward-pass normalization, potentially enabling normalization-free training. This approach provides a principled mechanism to align optimization geometry with the intrinsic gauge symmetries of deep networks, with practical implications for stability, initialization robustness, and depth scalability. By removing the dependence on arbitrary diagonal gauges, the UC adjoint offers a pathway to more robust learning across parameterizations and could reduce or eliminate reliance on normalization layers in certain settings.
Abstract
Deep neural networks constructed from linear maps and positively homogeneous nonlinearities (e.g., ReLU) possess a fundamental gauge symmetry: the network function is invariant to node-wise diagonal rescalings. However, standard gradient descent is not equivariant to this symmetry, causing optimization trajectories to depend heavily on arbitrary parameterizations. Prior work has proposed rescaling-invariant optimization schemes for positively homogeneous networks (e.g., path-based or path-space updates). Our contribution is complementary: we formulate the invariance requirement at the level of the backward adjoint/optimization geometry, which provides a simple, operator-level recipe that can be applied uniformly across network components and optimizer state. By replacing the Euclidean transpose with a Unit-Consistent (UC) adjoint, we derive UC gauge-consistent steepest descent and backprogation.