AlphaFuse: Learn ID Embeddings for Sequential Recommendation in Null Space of Language Embeddings

TL;DR

This work tackles the degradation of semantic spaces when mapping high-dimensional language embeddings to lower-dimensional ID embeddings in sequential recommendation. It introduces AlphaFuse, which decomposes language embeddings via SVD into semantic-rich row space and semantic-sparse null space, clips and standardizes subspaces, and learns ID embeddings only in the null space while freezing the semantic-rich components. The approach yields a semantic-anchored, parameter-efficient integration of language information that is model-agnostic and compatible with both discriminative and diffusion-based backbones, achieving improvements in cold-start and long-tail scenarios across three Amazon datasets. The results highlight the practical impact of preserving semantic information while incorporating collaborative signals without auxiliary modules, with code and datasets publicly available.

Abstract

Recent advancements in sequential recommendation have underscored the potential of Large Language Models (LLMs) for enhancing item embeddings. However, existing approaches face three key limitations: 1) the degradation of the semantic space when high-dimensional language embeddings are mapped to lower-dimensional ID embeddings, 2) the underutilization of language embeddings, and 3) the reliance on additional trainable parameters, such as an adapter, to bridge the gap between the semantic and behavior spaces. In this paper, we introduce AlphaFuse, a simple but effective language-guided learning strategy that addresses these challenges by learning ID embeddings within the null space of language embeddings. Specifically, we decompose the semantic space of language embeddings via Singular Value Decomposition (SVD), distinguishing it into a semantic-rich row space and a semantic-sparse null space. Collaborative signals are then injected into the null space, while preserving the rich semantics of the row space. AlphaFuse prevents degradation of the semantic space, integrates the retained language embeddings into the final item embeddings, and eliminates the need for auxiliary trainable modules, enabling seamless adaptation to any sequential recommendation framework. We validate the effectiveness and flexibility of AlphaFuse through extensive experiments on three benchmark datasets, including cold-start user and long-tail settings, showcasing significant improvements in both discriminative and diffusion-based generative sequential recommenders. Our codes and datasets are available at https://github.com/Hugo-Chinn/AlphaFuse.

Figures (5)

• Figure 1: Visualization of space decomposition.
• Figure 2: Overall framework of language-guided ID embedding learning strategies.
• Figure 3: Normalized singular values of language embeddings for the Movies Amazon2023. For concision, we omit plots of other datasets as they exhibit similar trends.
• Figure 4: The cumulative distribution of cosine similarity for item pairs using language embeddings in the Movies Amazon2023. Other datasets exhibit similar trends.
• Figure 5: Hyperparameter experiments for AlphaFuse: the left subfigures analyze the impact of final item and ID embeddings on SASRec-based AlphaFuse, while the right one evaluates the effect of semantic weight threshold on DreamRec-based AlphaFuse.