On-Device Large Language Models for Sequential Recommendation
Xin Xia, Hongzhi Yin, Shane Culpepper
TL;DR
On-device sequential recommendation requires strong privacy and low latency, but LLMs are too large for edge deployment. This work introduces OD-LLM, a task-adaptive compression framework that combines token covariance normalization, low-rank Singular Value Decomposition (SVD) with a rank chosen by $r=\left\lfloor \dfrac{M N\, CR}{M+N} \right\rfloor$, and progressive layerwise updates to preserve sequential signals. It builds on LC-Rec fine-tuning with RQ-VAE indexing and Sinkhorn-based uniform mapping, optimized via a joint loss that blends reconstruction, RQ, optimal transport, and multiple instruction-tuning objectives. Empirically, OD-LLM achieves comparable or better recommendation quality than a full model while reducing parameters by about 2x at CR=0.5 and delivering faster on-device inference, enabling private, real-time, edge-based LLM-assisted recommendations. This approach demonstrates practical viability for privacy-preserving, edge-friendly LLM deployments in recommender systems.
Abstract
On-device recommendation is critical for a number of real-world applications, especially in scenarios that have agreements on execution latency, user privacy, and robust functionality when internet connectivity is unstable or even impossible. While large language models (LLMs) can now provide exceptional capabilities that model user behavior for sequential recommendation tasks, their substantial memory footprint and computational overhead make the deployment on resource-constrained devices a high risk proposition. In this paper, we propose OD-LLM, the first task-adaptive compression framework explicitly designed to provide efficient and accurate on-device deployment of LLMs for sequential recommendation tasks. OD-LLM uniquely integrates two complementary compression strategies: a low-rank structural compression algorithm which uses Singular Value Decomposition (SVD) to significantly reduce parameter redundancy in the model, and a novel tokenization normalization technique that better complements the low-rank decomposition process being used. Additionally, to minimize any potential performance degradation when using higher compression ratios, a novel progressive alignment algorithm is used to iteratively refine the parameters required layerwise in the target model. Empirical evaluations conducted on sequential recommendation benchmarks show that OD-LLM exhibits no loss in effectiveness when compared to the original recommendation model, when the deployed model size is halved. These promising results demonstrate the efficacy and scalability of OD-LLM, making this novel solution a practical alternative for real-time, on-device solutions wishing to replace expensive, remotely executed LLMs.