Structured Knowledge Accumulation: The Principle of Entropic Least Action in Forward-Only Neural Learning

TL;DR

The paper reframes neural learning as a forward-only, continuous-time process governed by entropy dynamics, introducing the Structured Knowledge Accumulation (SKA) framework. It advances two core ideas: (i) reinterpreting the learning rate as a time step and uncovering a characteristic learning time, and (ii) defining the Tensor Net function as a criterion for the onset of structured learning, together with a variational formulation based on the Euler-Lagrange equation. The key contributions include a time-invariant interpretation of learning, a layer-wise knowledge-entropy coupling via the Tensor Net, and a principled entropic least-action perspective that yields natural stopping criteria and a self-solving dynamic. The work bridges computational learning with physical dynamical systems, offering a physically grounded, potentially hardware-friendly pathway for robust, biologically-inspired AI with explicit intrinsic timescales and information-theoretic stopping rules.

Abstract

This paper aims to extend the Structured Knowledge Accumulation (SKA) framework recently proposed by \cite{mahi2025ska}. We introduce two core concepts: the Tensor Net function and the characteristic time property of neural learning. First, we reinterpret the learning rate as a time step in a continuous system. This transforms neural learning from discrete optimization into continuous-time evolution. We show that learning dynamics remain consistent when the product of learning rate and iteration steps stays constant. This reveals a time-invariant behavior and identifies an intrinsic timescale of the network. Second, we define the Tensor Net function as a measure that captures the relationship between decision probabilities, entropy gradients, and knowledge change. Additionally, we define its zero-crossing as the equilibrium state between decision probabilities and entropy gradients. We show that the convergence of entropy and knowledge flow provides a natural stopping condition, replacing arbitrary thresholds with an information-theoretic criterion. We also establish that SKA dynamics satisfy a variational principle based on the Euler-Lagrange equation. These findings extend SKA into a continuous and self-organizing learning model. The framework links computational learning with physical systems that evolve by natural laws. By understanding learning as a time-based process, we open new directions for building efficient, robust, and biologically-inspired AI systems.

Figures (6)

• Figure 1: Comparison of entropy evolution patterns across different time step configurations. Despite different discretization granularity, the overall trajectories remain identical when $\eta \times K = 0.5$, demonstrating the time-invariant nature of SKA learning.
• Figure 2: Comparison of cosine evolution patterns across different time step configurations. Despite different discretization granularity, the overall trajectories remain identical when $\eta \times K = 0.5$, demonstrating the time-invariant nature of SKA learning.
• Figure 3: Knowledge Flow versus the Frobenius norm of the knowledge tensor for each layer. The color gradient represents training progression (darker blue points occur earlier in training). Red dots mark the entropy minima for each layer. The experiment is conducted within the valid characteristic time scale ($T = 0.5$).
• Figure 4: Knowledge Flow evolution over discrete steps K. The experiment is conducted within the valid characteristic time scale ($T = 0.5$), ensuring that knowledge flow convergence accurately represents structured knowledge accumulation completion.
• Figure 5: The red marks illustrate the entropy minima present for each layer. The green dots represent the zero-crossing points where $\int \mathbf{D}^{(l)} \, dz = \mathbf{H}^{(l)}$. Each layer shows different behaviors that correspond to its position in the hierarchy. The earlier layers have smooth transitions, whereas, the output layer (Layer 4) has a negative trajectory which is attributed to its classification task.