Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Laws of Learning Dynamics and the Core of Learners

Inkee Jung, Siu Cheong Lau

TL;DR

This work formalizes learning dynamics for single models and ensembles by defining entropy measures and two core laws: a conservation law linking learning effort to entropy change, and a monotone entropy decrease during learning. It extends to lifelong, modular ensembles via logifolds, cores, and memory formation, proposing an immunized two-generation architecture (IMM) that detects adversarial perturbations through ensemble entropy and adapts by incorporating a second-generation trained on perturbed data. Empirically, IMM improves accuracy on the union of clean and adversarial CIFAR-10 samples and reduces entropy compared to naive ensembles, with notable gains under strong perturbations and transfer-based attacks. The results suggest a practical framework for progressively expanding, domain-aware ensembles that preserve memory and robustness through entropy-driven core detection and generation transfer.

Abstract

We formulate the fundamental laws governing learning dynamics, namely the conservation law and the decrease of total entropy. Within this framework, we introduce an entropy-based lifelong ensemble learning method. We evaluate its effectiveness by constructing an immunization mechanism to defend against transfer-based adversarial attacks on the CIFAR-10 dataset. Compared with a naive ensemble formed by simply averaging models specialized on clean and adversarial samples, the resulting logifold achieves higher accuracy in most test cases, with particularly large gains under strong perturbations.

Laws of Learning Dynamics and the Core of Learners

TL;DR

This work formalizes learning dynamics for single models and ensembles by defining entropy measures and two core laws: a conservation law linking learning effort to entropy change, and a monotone entropy decrease during learning. It extends to lifelong, modular ensembles via logifolds, cores, and memory formation, proposing an immunized two-generation architecture (IMM) that detects adversarial perturbations through ensemble entropy and adapts by incorporating a second-generation trained on perturbed data. Empirically, IMM improves accuracy on the union of clean and adversarial CIFAR-10 samples and reduces entropy compared to naive ensembles, with notable gains under strong perturbations and transfer-based attacks. The results suggest a practical framework for progressively expanding, domain-aware ensembles that preserve memory and robustness through entropy-driven core detection and generation transfer.

Abstract

We formulate the fundamental laws governing learning dynamics, namely the conservation law and the decrease of total entropy. Within this framework, we introduce an entropy-based lifelong ensemble learning method. We evaluate its effectiveness by constructing an immunization mechanism to defend against transfer-based adversarial attacks on the CIFAR-10 dataset. Compared with a naive ensemble formed by simply averaging models specialized on clean and adversarial samples, the resulting logifold achieves higher accuracy in most test cases, with particularly large gains under strong perturbations.
Paper Structure (18 sections, 7 theorems, 37 equations, 1 figure, 2 tables)

This paper contains 18 sections, 7 theorems, 37 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Laws of Learning for a model
  3. Law of Learning for an ensemble
  4. Entropy-based lifelong ensemble learning
  5. Immunized systems against adversarial attacks
  6. Experimental setup
  7. Specialization, composing generations, and evaluation
  8. Compute environment.
  9. Results
  10. Ensemble entropy versus single-model entropy.
  11. Evaluation across environments and memory formation.
  12. Conclusion
  13. Proofs
  14. Proof of Proposition \ref{['prop:PD']}
  15. Proof of Theorem \ref{['thm:van_ent']}
  16. ...and 3 more sections

Key Result

Proposition 2.4

Figures (1)

  • Figure 1: Accuracy and coverage under entropy thresholding across domains. Solid curves show accuracies on $\mathcal{C}_\tau$ (black) and $\mathcal{C}_\tau^{c}$ (red). Dashed curves show coverages of $\mathcal{C}_\tau$ (orange) and $\mathcal{C}_\tau^{c}$ (blue). The vertical dashed line marks the selected $\tau$ for a validation accuracy target of $95.25\%$. Across domains, decreasing $\tau$ monotonically decreases core coverage while increasing core accuracy. Moreover, the core accuracy is consistently above the overall accuracy, whereas the out-of-core accuracy is consistently below it.

Theorems & Definitions (28)

  • Definition 2.1
  • Definition 2.2: Entropy of a model
  • Definition 2.3: Cross entropy between the truth and a model
  • Proposition 2.4: Conservation Law of Learning
  • proof
  • Definition 2.5
  • Theorem 2.6
  • proof
  • Definition 3.1
  • Definition 3.2: Entropy for ensemble
  • ...and 18 more