Confidence Preservation Property in Knowledge Distillation Abstractions
Dmitry Vengertsev, Elena Sherman
TL;DR
The paper investigates whether knowledge distillation, as implemented by TinyBERT, preserves the confidence (beyond accuracy) of a large teacher BERT model. It introduces a pairwise input-specific confidence measure, defines a global preservation criterion φ_cnf via the statistic $\boldsymbol{\text{σ}}(X^{train}) \le \kappa$, and derives a theoretical bound linking φ_cnf to distillation losses with $\kappa \le \gamma \sqrt{\beta / (|X^{train}|(1-\alpha))}$; with TinyBERT defaults this simplifies to $\boldsymbol{\text{σ}}(X^{train}) < \sqrt{\beta / |X^{train}|}$. Empirically, φ_cnf is not uniformly preserved across six GLUE tasks: the 6-layer TinyBERT ($S_{6L}$) maintains the property for three tasks but not for the other three, while the 4-layer version ($S_{4L}$) fails entirely. By tuning distillation hyperparameters, especially for the prediction and intermediate layers, the authors show that φ_cnf can be restored for the failed tasks without meaningful accuracy degradation, highlighting the practical value of considering confidence preservation in distillation design and tuning.
Abstract
Social media platforms prevent malicious activities by detecting harmful content of posts and comments. To that end, they employ large-scale deep neural network language models for sentiment analysis and content understanding. Some models, like BERT, are complex, and have numerous parameters, which makes them expensive to operate and maintain. To overcome these deficiencies, industry experts employ a knowledge distillation compression technique, where a distilled model is trained to reproduce the classification behavior of the original model. The distillation processes terminates when the distillation loss function reaches the stopping criteria. This function is mainly designed to ensure that the original and the distilled models exhibit alike classification behaviors. However, besides classification accuracy, there are additional properties of the original model that the distilled model should preserve to be considered as an appropriate abstraction. In this work, we explore whether distilled TinyBERT models preserve confidence values of the original BERT models, and investigate how this confidence preservation property could guide tuning hyperparameters of the distillation process.