Learning to Reason: Temporal Saliency Distillation for Interpretable Knowledge Transfer

TL;DR

Knowledge distillation in time series often transfers only predictive accuracy and neglects the teacher's reasoning, hindering safe teacher substitution. Temporal Saliency Distillation (TSD) extends logit transfer by extracting temporal saliency from the teacher via perturbation-based distributional shifts and training the student to match these saliency maps using a normalized loss $L_{\mathrm{TSKD}}$. The approach is model-agnostic and requires no architecture changes, with a distillation objective that combines $L_{\mathrm{KD}}$ and $L_{\mathrm{TSKD}}$, including temperature scaling and perturbation-based KL divergences. Experiments on 28 UCR datasets show that TSD improves generalization and fidelity relative to state-of-the-art KD methods, preserves teacher-level interpretability as measured by saliency-map similarity, and remains robust to limited data and noise. Overall, TSD introduces a new paradigm for interpretable KD in time series, enabling safer and more trustworthy deployment of compressed models in real-world settings.

Abstract

Knowledge distillation has proven effective for model compression by transferring knowledge from a larger network called the teacher to a smaller network called the student. Current knowledge distillation in time series is predominantly based on logit and feature aligning techniques originally developed for computer vision tasks. These methods do not explicitly account for temporal data and fall short in two key aspects. First, the mechanisms by which the transferred knowledge helps the student model learning process remain unclear due to uninterpretability of logits and features. Second, these methods transfer only limited knowledge, primarily replicating the teacher predictive accuracy. As a result, student models often produce predictive distributions that differ significantly from those of their teachers, hindering their safe substitution for teacher models. In this work, we propose transferring interpretable knowledge by extending conventional logit transfer to convey not just the right prediction but also the right reasoning of the teacher. Specifically, we induce other useful knowledge from the teacher logits termed temporal saliency which captures the importance of each input timestep to the teacher prediction. By training the student with Temporal Saliency Distillation we encourage it to make predictions based on the same input features as the teacher. Temporal Saliency Distillation requires no additional parameters or architecture specific assumptions. We demonstrate that Temporal Saliency Distillation effectively improves the performance of baseline methods while also achieving desirable properties beyond predictive accuracy. We hope our work establishes a new paradigm for interpretable knowledge distillation in time series analysis.

Figures (6)

• Figure 1: An overview of the proposed TSD. The student network learns the target task by minimizing the classification loss while mimicking the temporal saliency observed by the teacher network.
• Figure 2: TSD transfers the right reasoning. (Left) Mean signals for the Coffee dataset for each class, the width representing the standard deviation. Compared to Robusta, Arabica beans have lower caffeine and chlorogenic acid content, contributing to their finer taste. The shaded area provides information about the caffeine content, which exhibits class-discriminative power compared to other regions briandet1996discriminationdelaney2021instance. (Center) Contribution of each input index to the class prediction for Robusta using SHAP analysis lindberg2017unified. Red indicates a positive contribution, and blue indicates a negative one. The TSD student exhibits input feature contributions consistent with the teacher, even in regions that are less class-discriminative. (Right) Same visualization for Arabica.
• Figure 3: TSD transfer properties beyond generalization. (Left) TSD enables student models to achieve both good generalization and good fidelity. Generalization is measured by AUC-PRC, and fidelity by test agreement on seven multiclass UCR datasets. (Right) TSD transfers the interpretability of the teacher model. The discrepancy between teacher and student saliency maps is measured using MSE. Similarity improves for both gradient-based and perturbation-based saliency maps.
• Figure 4: Evaluation of test fidelity across seven multi-class UCR datasets: (left) average top-1 agreement; (center) average predictive KL divergence. (Right) Train-test fidelity across various expansion ratios of the distillation set with GAN generated synthetic data.
• Figure 5: (Left) Performance of students on seven multi-class UCR datasets where training set is reduced at various ratios. (Right) Performance (average AUC-PRC) degradation under varying adversarial noise levels. Both teacher-specific and student-specific FGSM noise are considered.