Recursive Meta-Distillation: An Axiomatic Framework for Iterative Knowledge Refinement

TL;DR

This work advances recursive knowledge distillation by formulating an axiomatic, operator-theoretic framework that anchors each generation to a base teacher. It proves a contraction property in KL divergence, yielding geometric convergence to base-teacher distributions and a globally attractive fixed point, independent of architectural or optimization details. The approach clarifies how anchoring mitigates drift, provides a bias-variance perspective for iterative refinement, and delineates where and how previous methods differ. These theoretical guarantees offer a principled foundation for stable, multi-generation distillation under capacity constraints, with practical convergence criteria and directions for extension.

Abstract

Recent work in probability-domain knowledge distillation has established axiomatic frameworks for temperature scaling, multi-teacher aggregation, and bias-variance trade-offs in single-stage settings. However, the mathematical behavior of recursive or multi-generation distillation remains poorly understood, with prior approaches relying primarily on empirical heuristics. In this work, we introduce an axiomatic and operator-theoretic framework for recursive meta-distillation, formalizing iterative knowledge distillation as a sequence of probability-distribution operators with explicit anchoring to base teachers. We define structural axioms for valid meta-teacher construction and prove the existence of non-trivial operator families satisfying these axioms without specifying particular algorithms or loss functions. Under mild realizability and convexity assumptions, we show that anchored recursive distillation induces contraction in KL divergence, yielding geometric convergence to base teacher distributions and a unique, globally attractive fixed point. The contribution is foundational rather than algorithmic: the framework characterizes when recursive distillation is mathematically well-posed and convergent rather than error-accumulating, independent of model architecture, optimization details, or specific operator instantiations. These results provide a theoretical basis for understanding stability, bias-variance behavior, and failure modes in iterative and multi-teacher distillation under capacity constraints.

Figures (1)

• Figure 1: Schematic comparison of error evolution across generations under recursive distillation. Without teacher anchoring ($\alpha = 0$), approximation errors accumulate linearly, producing unbounded drift. With anchoring ($\alpha > 0$), the recursive operator becomes contractive, yielding geometric decay toward the base teacher.

Theorems & Definitions (36)

• Definition 2.1: Recursive Distillation Environment
• Remark 2.2: Positivity Assumption
• Remark 2.3: Temperature Scope Clarification
• Definition 3.1: Meta-Teacher Construction Operator
• Remark 3.2: Sufficient Anchoring Condition
• Remark 3.3: Role of the Anchor Weight $\alpha$
• Theorem 3.4: Existence of Conforming Operators
• proof
• Theorem 3.5: Non-Uniqueness
• proof