Implicit bias as a Gauge correction: Theory and Inverse Design

TL;DR

The paper introduces a geometric mechanism for implicit bias in SGD: continuous parameter symmetries induce a gauge correction when dynamics are projected to the quotient space $\bar{\Theta}$, biasing solutions toward regions with smaller orbit volume. It formalizes this with a stochastic differential equation framework, defines the Faddeev–Popov and orbit Gram matrices, and shows that the stationary distribution on a gauge slice acquires a $\log\det G_\chi$ correction. It then develops constructive inverse-design principles, showing how scalar and matrix factorizations, via Abelian and non-Abelian symmetries, yield targeted biases such as spectral sparsity and total-variation tendencies, and validates these with numerical experiments on toy data and with several architecture exemplars. The framework unifies known implicit biases under a common geometric lens and offers a principled route to engineer inductive biases by symmetry design, with potential applications to ill-posed inverse problems and beyond. The work also connects to prior symmetry- and thermodynamics-informed perspectives while providing new quantitative tools (gauge corrections, balanced gauges) to predict and realize bias in complex models.

Abstract

A central problem in machine learning theory is to characterize how learning dynamics select particular solutions among the many compatible with the training objective, a phenomenon, called implicit bias, which remains only partially characterized. In the present work, we identify a general mechanism, in terms of an explicit geometric correction of the learning dynamics, for the emergence of implicit biases, arising from the interaction between continuous symmetries in the model's parametrization and stochasticity in the optimization process. Our viewpoint is constructive in two complementary directions: given model symmetries, one can derive the implicit bias they induce; conversely, one can inverse-design a wide class of different implicit biases by computing specific redundant parameterizations. More precisely, we show that, when the dynamics is expressed in the quotient space obtained by factoring out the symmetry group of the parameterization, the resulting stochastic differential equation gains a closed form geometric correction in the stationary distribution of the optimizer dynamics favoring orbits with small local volume. We compute the resulting symmetry induced bias for a range of architectures, showing how several well known results fit into a single unified framework. The approach also provides a practical methodology for deriving implicit biases in new settings, and it yields concrete, testable predictions that we confirm by numerical simulations on toy models trained on synthetic data, leaving more complex scenarios for future work. Finally, we test the implicit bias inverse-design procedure in notable cases, including biases toward sparsity in linear features or in spectral properties of the model parameters.

Figures (10)

• Figure 1: Geometry of parameter symmetries and induced implicit bias.(a) The parameter space $\Theta$ contains symmetry orbits $\mathcal{O}_\theta$ (loops) where the loss $L(\theta)$ is constant. (b) The projection onto the quotient space $\bar{\Theta}$ maps entire orbits to single points. The Jacobian of this map introduces a volume factor $\sqrt{\det G_\chi(\theta)}$. (c) This geometric correction acts as an additional potential in the effective loss $L_{\mathrm{eff}}$, creating a statistical preference for solutions with smaller orbit volumes (implicit bias) without explicit regularization.
• Figure 2: Empirical radial density from simulated $d=10$ Langevin dynamics versus theoretical predictions with and without the orbit-volume (gauge) correction. The gauge-corrected prediction $\rho_\mathcal{S}(r)\propto r^{d-1}e^{-\beta\tilde{\ell}(r)}$ matches the empirical distribution, whereas the naive prediction does not.
• Figure 3: Top As training progresses, the non-biased model fails to achieve significant improvement in its loss over the test-set, while its training loss sharply decreases -- a clean signal of overfitting. The biased model instead experiences a steady improvement in both losses, with its test loss soon overcoming that of the vanilla model -- thus attaining far better generalization. Middle Recovered Fourier coefficients. Even though spectral components of the vanilla model partially identify the dominant modes from ground truth, their expression is insufficient and significant spectral noise is present. The biased model closely adheres to the spectral content of the ground truth. Bottom Analysis of the fit in time domain reveals that the vanilla model operates in purely-interpolating regime across training data, i.e. essentially matching the ground truth only on the training dataset and nowhere else. On the other hand, a close match between the biased model and ground truth is achieved almost everywhere. The comparison of the $\ell_1$ spectral norms across the biased model ($4.89$), the vanilla model ($10.72$), and ground truth ($4.68$) corroborate the theoretical prediction of a spectral sparsification implicit bias even in the lack of explicit regularization.
• Figure 4: Top The vanilla, unbiased and not regularized, model fails to capture effectively the underlying piecewise-constant structure of the training signal and the predictor exhibits a significant noisy behaviour. The biased model, obtained via the inverse-designed implicit bias on total variation, and no additional regularization, closely adheres to the target signal and exhibits low local noise. Middle, Bottom Both the vanilla and biased models rapidly converge to almost-zero loss over the training signal, and -- correspondingly -- to stable amounts of total variation. The resulting total variation in the vanilla model is more than six times that of the biased model. Note that the vanilla model, on the other hand, still achieves statistical error minimization -- though at the cost of complete loss of structure in the recovered signal.
• Figure 5: Top Train/test learning curves for the three parameterizations. Although all models can drive the training MSE down, the naive and scalar factorization model exhibits markedly worse generalization. The matrix factorization instead achieves a substantially lower test error and stabilizes earlier, indicating that the induced geometric bias is aligned with the ground-truth structure. Middle Singular values of the learned weight matrix $W$. The ground truth is strongly low-rank, with a pronounced spectral gap ($i$ is the channel index). The naive model spreads energy across many singular directions (consistent with an approximately isotropic, Frobenius-type bias), and the scalar factorization has a similar behavior. In contrast, the matrix factorization model closely matches the true singular spectrum, recovering the correct effective rank. Bottom Channel reconstructions on a dense evaluation grid. The matrix factorization model tracks the ground truth across channels uniformly, reflecting recovery of the shared low-dimensional channel subspace. The naive and scalar factorization model reconstructions are poorer.

Theorems & Definitions (8)

• Theorem 1: Gauge-fixed stationary distribution on a slice
• Proposition 1: Constraint/orbit Gram relation
• Proposition 2: Scalar factorization induces logarithmic features sparsity
• Proposition 3: Matrix factorization induces logarithmic spectral sparsity and norm balancing
• Remark 1: Polynomial and homogeneous non-linearities
• Remark 2: Smooth non-linearities and approximate spectral bias
• proof
• proof