LLM-based Knowledge Pruning for Time Series Data Analytics on Edge-computing Devices

TL;DR

The paper tackles the problem of limited generalization in time-series models on resource-constrained devices by introducing Knowledge Pruning (KP), a paradigm that prunes redundant world knowledge from large language models (LLMs) and distills only the pertinent knowledge into a lightweight target model without loading the LLM during training or inference. KP builds a knowledge prompt set (KPS) to generate knowledge anchor points (KAPs) via a pre-trained LLM, then uses a two-layer alignment module and metric learning to transfer this prior knowledge through a distillation loss, with an additional Anchor Voting Scheme (AVS) enabling regression by producing continuous outputs from discrete anchor scores. The approach is validated on edge-focused time-series tasks, specifically human activity recognition (HAR) and remaining useful life (RUL) prediction, across eight benchmarks, where KP achieves up to 19.7% RMSE improvement and up to 13.7% F1 improvement, while drastically reducing computational demands compared to LLM-based baselines. Overall, KP demonstrates that only task-relevant (pertinent) LLM knowledge needs to be transferred for effective time-series analytics on edge devices, enabling practical deployment of LLM-informed methods in constrained environments.

Abstract

Limited by the scale and diversity of time series data, the neural networks trained on time series data often overfit and show unsatisfacotry performances. In comparison, large language models (LLMs) recently exhibit impressive generalization in diverse fields. Although massive LLM based approaches are proposed for time series tasks, these methods require to load the whole LLM in both training and reference. This high computational demands limit practical applications in resource-constrained settings, like edge-computing and IoT devices. To address this issue, we propose Knowledge Pruning (KP), a novel paradigm for time series learning in this paper. For a specific downstream task, we argue that the world knowledge learned by LLMs is much redundant and only the related knowledge termed as "pertinent knowledge" is useful. Unlike other methods, our KP targets to prune the redundant knowledge and only distill the pertinent knowledge into the target model. This reduces model size and computational costs significantly. Additionally, different from existing LLM based approaches, our KP does not require to load the LLM in the process of training and testing, further easing computational burdens. With our proposed KP, a lightweight network can effectively learn the pertinent knowledge, achieving satisfactory performances with a low computation cost. To verify the effectiveness of our KP, two fundamental tasks on edge-computing devices are investigated in our experiments, where eight diverse environments or benchmarks with different networks are used to verify the generalization of our KP. Through experiments, our KP demonstrates effective learning of pertinent knowledge, achieving notable performance improvements in regression (19.7% on average) and classification (up to 13.7%) tasks, showcasing state-of-the-art results.

Figures (4)

• Figure 1: The world knowledge learned by LLMs. Different parts of the knowledge of LLMs may contribute differently on diverse tasks. For a specific task, the world knowledge in LLMs is actually redundant and only the related knowledge termed as pertinent knowledge is useful. Our proposed Knowledge Pruning (KP) aims to prune the redundant knowledge and effectively transfer the pertinent knowledge to the target model, significantly reducing computation cost while retaining satisfactory performances.
• Figure 2: The pipeline of our knowledge pruning (KP). Our KP coonsists of two stages: pre-processing and training stage. For a specific task, a knowledge prompt set is firstly produced, where these prompts are forwarded into a pre-trained LLM to obtain the corresponding language embeddings. Then, these embeddings are used as knowledge anchor points to estimate the pertinent knowledge and prune redundant knowledge of the LLM. After that, in the training stage, these knowledge anchor points are regarded as prior knowledge. The prior knowledges are transfered to the target mode via knowledge distillation.
• Figure 3: Expriements on RUL task with different $\tau$ values on FD004.
• Figure 4: Expriements on HAR task with different $\tau$ values.