Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregation

TL;DR

This work addresses multi-teacher knowledge distillation by proposing an axiomatic, operator-theoretic framework for probability-domain aggregation that does not fix a single formula. It defines five core axioms (A1–A5) and proves the existence, and non-uniqueness, of operator families $G$ that satisfy them, enabling operator-agnostic guarantees. The authors show that, under assumptions such as linear-in-weights (Assumption L), the ensemble achieves variance reduction, Jensen-type bounds that favor mixture objectives, log-loss improvements over average teacher performance, and safety attenuation for heterogeneous specializations. The framework provides principled guidance for combining diverse frontier models, with practical design principles for per-teacher temperatures, weights, and mixture loss, while keeping a broad, implementation-flexible foundation.

Abstract

Building on the probability-domain distillation framework of Sparse-KD, we develop an axiomatic, operator-theoretic framework for multi-teacher ensemble knowledge distillation. Rather than prescribing a specific aggregation formula, we define five core axioms governing valid knowledge aggregation operators, encompassing convexity, positivity, continuity, weight monotonicity, and temperature coherence. We prove the existence and non-uniqueness of operator families satisfying these axioms, establishing that multiple distinct aggregation mechanisms conform to the same foundational principles. Within this framework, we establish operator-agnostic guarantees showing that multi-teacher aggregation reduces both stochastic variance and systematic supervisory bias under heterogeneous teachers, while providing Jensen-type bounds, log-loss guarantees, and safety attenuation properties. For aggregation operators linear in teacher weights, we further establish classical ensemble variance-reduction results under standard independence assumptions, with extensions to correlated-error regimes. The framework provides theoretical grounding for multi-teacher distillation from diverse frontier models while admitting multiple valid implementation strategies.

Theorems & Definitions (30)

• Definition 2.1: Multi-Teacher Setting
• Definition 2.2: Multi-Teacher Aggregation Operator
• Example 2.3: Illustrative Only
• Theorem 3.1: Existence of Conforming Operators
• proof
• Theorem 3.2: Non-Uniqueness
• proof
• Remark 3.3: Operator Non-Identifiability
• Definition 4.1: Cross-Teacher Variance
• Theorem 4.2: Ensemble Variance Reduction