TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation

TL;DR

TD3 addresses the challenge of distilling large, discrete sequential user-item data for recommender systems by introducing a Tucker decomposition-based synthetic sequence summary. It factorizes the synthetic data into four components (U, T, V, G), sharing the item factor with the embedding to handle large item sets, and couples this with an enhanced bi-level optimization that includes augmented learner training, feature space alignment, and RaT-BPTT for efficiency. The method achieves comparable or superior performance to full-data training with significantly smaller synthetic data, and demonstrates cross-architecture generalization across SASRec, GRU4Rec, NARM, and BERT4Rec, underscoring the data-centric paradigm for sequential recommendations. These results suggest substantial practical impact in reducing training time and resource use while maintaining or improving recommendation quality, with future work aimed at scaling and cross-domain transfer.

Abstract

In the era of data-centric AI, the focus of recommender systems has shifted from model-centric innovations to data-centric approaches. The success of modern AI models is built on large-scale datasets, but this also results in significant training costs. Dataset distillation has emerged as a key solution, condensing large datasets to accelerate model training while preserving model performance. However, condensing discrete and sequentially correlated user-item interactions, particularly with extensive item sets, presents considerable challenges. This paper introduces \textbf{TD3}, a novel \textbf{T}ucker \textbf{D}ecomposition based \textbf{D}ataset \textbf{D}istillation method within a meta-learning framework, designed for sequential recommendation. TD3 distills a fully expressive \emph{synthetic sequence summary} from original data. To efficiently reduce computational complexity and extract refined latent patterns, Tucker decomposition decouples the summary into four factors: \emph{synthetic user latent factor}, \emph{temporal dynamics latent factor}, \emph{shared item latent factor}, and a \emph{relation core} that models their interconnections. Additionally, a surrogate objective in bi-level optimization is proposed to align feature spaces extracted from models trained on both original data and synthetic sequence summary beyond the naïve performance matching approach. In the \emph{inner-loop}, an augmentation technique allows the learner to closely fit the synthetic summary, ensuring an accurate update of it in the \emph{outer-loop}. To accelerate the optimization process and address long dependencies, RaT-BPTT is employed for bi-level optimization. Experiments and analyses on multiple public datasets have confirmed the superiority and cross-architecture generalizability of the proposed designs. Codes are released at https://github.com/USTC-StarTeam/TD3.

Figures (4)

• Figure 1: Comparison of the data-centric recommender system through the lens of dataset distillation approach with the traditional model-centric recommender system. The key difference lies in their distinct optimization objectives.
• Figure 2: An illustration of Tucker decomposition. The left part shows a three-dimensional synthetic sequence summary, with the third dimension representing the probability distribution over the entire item set. The right part illustrates the tucker decomposition, composed of a core tensor and factor matrices. The user tensor, temporal tensor, and core tensor are the parameters to be learned, while the item tensor shares values with the trained item embedding table.
• Figure 3: Illustration of TD3. In step 1, the learner is trained to get the best checkpoint for feature space alignment and item embedding for the shared item latent factor, and checkpoints from early epochs are saved to form a pool for initialization in each outer-loop. Step 2 visualizes a single outer-loop step, where the meta-gradient is computed from both the test loss and the feature space alignment loss. Step 3 demonstrates that the distilled summary enables training of other networks with similar performance to models trained on real data, while significantly reducing training time and memory usage.
• Figure 4: Illustration of the performance comparison of the TD3 against: Farzi, random sampling and longest sampling.