MemKD: Memory-Discrepancy Knowledge Distillation for Efficient Time Series Classification

TL;DR

MemKD tackles efficient time-series classification by tailoring knowledge distillation to the memory dynamics of RNNs. It defines memory-change $\Delta \boldsymbol{h}^{t,z} = \boldsymbol{h}^{(t+z)} - \boldsymbol{h}^{(t)}$ and distills knowledge via a normalized, subsequence-aware loss $L_{MemKD}$ based on Smooth L1 between teacher and student memory responses. On 12 UCR datasets, MemKD reduces model size by about $500\times$ while delivering performance comparable to the teacher and outperforming existing KD methods. This memory-centric approach enables compact, high-accuracy time-series classifiers suitable for edge and real-time deployment.

Abstract

Deep learning models, particularly recurrent neural networks and their variants, such as long short-term memory, have significantly advanced time series data analysis. These models capture complex, sequential patterns in time series, enabling real-time assessments. However, their high computational complexity and large model sizes pose challenges for deployment in resource-constrained environments, such as wearable devices and edge computing platforms. Knowledge Distillation (KD) offers a solution by transferring knowledge from a large, complex model (teacher) to a smaller, more efficient model (student), thereby retaining high performance while reducing computational demands. Current KD methods, originally designed for computer vision tasks, neglect the unique temporal dependencies and memory retention characteristics of time series models. To this end, we propose a novel KD framework termed Memory-Discrepancy Knowledge Distillation (MemKD). MemKD leverages a specialized loss function to capture memory retention discrepancies between the teacher and student models across subsequences within time series data, ensuring that the student model effectively mimics the teacher model's behaviour. This approach facilitates the development of compact, high-performing recurrent neural networks suitable for real-time, time series analysis tasks. Our extensive experiments demonstrate that MemKD significantly outperforms state-of-the-art KD methods. It reduces parameter size and memory usage by approximately 500 times while maintaining comparable performance to the teacher model.

Figures (1)

• Figure 1: An overview of the proposed knowledge distillation framework. The student learns target task by minimizing the classification loss while mimicking the memory retention of the teacher network using intermediate memory-maps.