An Equivariance Toolbox for Learning Dynamics
Yongyi Yang, Liu Ziyin
TL;DR
This work tackles how symmetry and equivariance structure constrain learning dynamics by introducing a general equivariance toolbox that produces coupled first- and second-order identities for parameterized models with outputs transforming under (H,G). The method defines characteristic directions X and Y in parameter and output spaces and derives universal first- and second-order relations for continuous equivariances, as well as fixed-point identities for discrete symmetries, unifying Noether-type conservation laws with homogeneity-based implicit bias. The key contributions include (i) extending Noether analysis to second-order Hessian structure, (ii) clarifying how Hessian action and eigenstructure align with transformation directions, (iii) linking parameter-space curvature to output-space geometry in last-layer and mirror-symmetry settings, and (iv) providing concrete results across homogeneity, continuous symmetries, and discrete symmetries. The framework offers new theoretical insight into optimization geometry, explaining phenomena such as low-dimensional gradient subspaces, progressive sharpening, and Hessian degeneracies observed in modern neural networks, with broad potential to inform training dynamics and architectural design.
Abstract
Many theoretical results in deep learning can be traced to symmetry or equivariance of neural networks under parameter transformations. However, existing analyses are typically problem-specific and focus on first-order consequences such as conservation laws, while the implications for second-order structure remain less understood. We develop a general equivariance toolbox that yields coupled first- and second-order constraints on learning dynamics. The framework extends classical Noether-type analyses in three directions: from gradient constraints to Hessian constraints, from symmetry to general equivariance, and from continuous to discrete transformations. At the first order, our framework unifies conservation laws and implicit-bias relations as special cases of a single identity. At the second order, it provides structural predictions about curvature: which directions are flat or sharp, how the gradient aligns with Hessian eigenspaces, and how the loss landscape geometry reflects the underlying transformation structure. We illustrate the framework through several applications, recovering known results while also deriving new characterizations that connect transformation structure to modern empirical observations about optimization geometry.