Backpropagation as Physical Relaxation: Exact Gradients in Finite Time

TL;DR

This work reframes backpropagation as the exact finite-time trace of a continuous physical relaxation, addressing the mismatch between symbolic gradient computation and physical dynamics. It introduces Dyadic Backpropagation (DBP), a variational framework that doubles the state space and uses a global energy on $(\boldsymbol{x},\boldsymbol{z})$ to encode forward and gradient information, enabled by a strictly lower-triangular, nilpotent weight structure. The authors prove that unit-step Euler discretization recovers standard backpropagation exactly in $2L$ steps for an $L$-layer network, without requiring weight symmetry or asymptotic limits, and demonstrate robust gradient fidelity and identical learning dynamics on CIFAR-10 across multiple step sizes. The approach has practical impact for analog and neuromorphic substrates where continuous dynamics are native and provides a theoretical bridge between physical relaxation processes and exact gradient computation, with potential connections to multi-compartment cortical models.

Abstract

Backpropagation, the foundational algorithm for training neural networks, is typically understood as a symbolic computation that recursively applies the chain rule. We show it emerges exactly as the finite-time relaxation of a physical dynamical system. By formulating feedforward inference as a continuous-time process and applying Lagrangian theory of non-conservative systems to handle asymmetric interactions, we derive a global energy functional on a doubled state space encoding both activations and sensitivities. The saddle-point dynamics of this energy perform inference and credit assignment simultaneously through local interactions. We term this framework ''Dyadic Backpropagation''. Crucially, we prove that unit-step Euler discretization, the natural timescale of layer transitions, recovers standard backpropagation exactly in precisely 2L steps for an L-layer network, with no approximations. Unlike prior energy-based methods requiring symmetric weights, asymptotic convergence, or vanishing perturbations, our framework guarantees exact gradients in finite time. This establishes backpropagation as the digitally optimized shadow of a continuous physical relaxation, providing a rigorous foundation for exact gradient computation in analog and neuromorphic substrates where continuous dynamics are native.

Figures (8)

• Figure 1: Generalization Performance. Test accuracy evolution for varying step sizes $\eta$ matches the BP baseline (dashed black), reaching $\sim93\%$.
• Figure 2: Empirical Validation of Dynamics. (left) Relative gradient error shows accurate recovery of $\nabla_{BP}$. (center) Relaxation steps converge to the theoretical limit $2L=18$ as $\eta\rightarrow1$. (right) Log-misalignment heatmap ($\eta=0.75$) plots $\log_{10}(1 - \text{CosSim})$ for each layer $\text{L}_i$. Values hover between $-6$ and $-8$, bounded by IEEE 754 Float32 machine precision ($\approx 10^{-7}$), rendering relaxation gradients basically indistinguishable from BP.
• Figure 3: Training Loss Consistency. The decay of training loss for the relaxation algorithm matches exactly across all step sizes. Axes indicate Epochs (0--100) versus Loss (log scale).
• Figure 4: Gradient Norm Fidelity. The ratio between the norms of relaxation gradients and backpropagation gradients remains centered at 1.0, with deviations smaller than $10^{-4}$. Legend indicates step sizes $\eta=0.25$ through $\eta=1.0$.
• Figure 5: Directional Alignment. The cosine misalignment ($1-\cos \theta$) between the relaxation algorithm and standard BP.

Theorems & Definitions (5)

• Lemma 3.1: Nilpotency of the Global Weight Matrix
• proof
• Lemma 4.1: Forward Layer Freezing
• proof
• Theorem 4.2: Autonomous Finite-Time Convergence