Structured Knowledge Accumulation: An Autonomous Framework for Layer-Wise Entropy Reduction in Neural Learning

TL;DR

The paper proposes Structured Knowledge Accumulation (SKA), a forward-only, gradient-free framework that redefines entropy as a continuous, layer-wise measure of knowledge alignment across a neural network. By deriving a persistent link between entropy reduction and the emergent sigmoid activation, SKA provides a biologically plausible mechanism for learning without backpropagation and enables independent layer-level optimization. Key results include a formal equivalence between SKA entropy and Shannon entropy under sigmoid decision probabilities, a fundamental law of entropy reduction for continuous and discrete dynamics, and a tensor-based implementation that demonstrates layer-wise entropy convergence, cosine alignment, and evolving decision probabilities. The approach offers scalable, interpretable learning suitable for resource-constrained and parallel computing environments, with potential impact on edge AI, neuroscience-inspired architectures, and real-time processing.

Abstract

We introduce the Structured Knowledge Accumulation (SKA) framework, which reinterprets entropy as a dynamic, layer-wise measure of knowledge alignment in neural networks. Instead of relying on traditional gradient-based optimization, SKA defines entropy in terms of knowledge vectors and their influence on decision probabilities across multiple layers. This formulation naturally leads to the emergence of activation functions such as the sigmoid as a consequence of entropy minimization. Unlike conventional backpropagation, SKA allows each layer to optimize independently by aligning its knowledge representation with changes in decision probabilities. As a result, total network entropy decreases in a hierarchical manner, allowing knowledge structures to evolve progressively. This approach provides a scalable, biologically plausible alternative to gradient-based learning, bridging information theory and artificial intelligence while offering promising applications in resource-constrained and parallel computing environments.

Figures (6)

• Figure 1: Layer entropy reduction for a given step $k$. The entropy level is represented by color intensity, darker blue corresponds to lower entropy. For illustration purpose only, entropy progressively decreases from Layer 1 to Layer 4. Each layer lowers entropy on a local level by coordinating knowledge vector $\mathbf{z}^{(l)}_k$ with decision change vector $\Delta \mathbf{D}^{(l)}_k$ as measured by $\cos(\theta^{(l)}_k)$.
• Figure 2: Entropy transformation over layers in an SKA neural network. Layers 2, 3, and 4 achieve a common entropy equilibrium $K = 49$. Layer 1 converges to this value, albeit at a more gradual rate.
• Figure 3: Cosine alignment evolution in SKA. The stabilization of the $\cos(\theta_k^{(l)})$ over steps indicates organized knowledge alignment across the layers.
• Figure 4: Change in output decision probabilities over forward steps in SKA. Distinctively from other models, SKA improves class distinctiveness incrementally without disrupting defined probability structures.
• Figure 5: Frobenius norm evolution of the knowledge tensors $\mathbf{z}^{(l)}$ throughout the layers during single-pass SKA training. It is often the case that the layer with the greater norm will have stronger activations because the updates are only being done in a forward manner and are more focused on local entropy minimization.