Are ID Embeddings Necessary? Whitening Pre-trained Text Embeddings for Effective Sequential Recommendation

TL;DR

The paper addresses whether ID embeddings are necessary for sequential recommendation by showing that pre-trained text embeddings are anisotropic and that whitening them yields strong text-only models (WhitenRec) that can rival or surpass ID-based approaches. To avoid losing semantic structure from whitening, the authors introduce WhitenRec+, an ensemble of fully whitened and relaxed whitened representations that preserves more information while maintaining isotropy. Through theoretical analyses of uniformity, alignment, conditioning, and information reconstruction, and extensive experiments on four public datasets under warm- and cold-start settings, WhitenRec and WhitenRec+ outperform state-of-the-art baselines, while reducing model complexity. The work demonstrates practical benefits for text-based sequential recommendation, including improved cold-start performance and transferability of text-derived item representations, with potential efficiency advantages in real-world deployment.

Abstract

Recent sequential recommendation models have combined pre-trained text embeddings of items with item ID embeddings to achieve superior recommendation performance. Despite their effectiveness, the expressive power of text features in these models remains largely unexplored. While most existing models emphasize the importance of ID embeddings in recommendations, our study takes a step further by studying sequential recommendation models that only rely on text features and do not necessitate ID embeddings. Upon examining pretrained text embeddings experimentally, we discover that they reside in an anisotropic semantic space, with an average cosine similarity of over 0.8 between items. We also demonstrate that this anisotropic nature hinders recommendation models from effectively differentiating between item representations and leads to degenerated performance. To address this issue, we propose to employ a pre-processing step known as whitening transformation, which transforms the anisotropic text feature distribution into an isotropic Gaussian distribution. Our experiments show that whitening pre-trained text embeddings in the sequential model can significantly improve recommendation performance. However, the full whitening operation might break the potential manifold of items with similar text semantics. To preserve the original semantics while benefiting from the isotropy of the whitened text features, we introduce WhitenRec+, an ensemble approach that leverages both fully whitened and relaxed whitened item representations for effective recommendations. We further discuss and analyze the benefits of our design through experiments and proofs. Experimental results on three public benchmark datasets demonstrate that WhitenRec+ outperforms state-of-the-art methods for sequential recommendation.

Figures (8)

• Figure 1: Overall Framework of presenting four variations of sequential recommendation methods, which are SASRec$^{ID}$, SASRec$^{T}$, WhitenRec, and WhitenRec+. Each method is composed of three components, including an item encoder, a sequence encoder, and the preference prediction layer.
• Figure 2: Normalized singular values of item text embeddings for Arts. For concision, we omit plots of other datasets as they exhibit similar trends.
• Figure 3: t-SNE plots of item text embeddings under different settings for Arts.
• Figure 4: CDF of item pairs.
• Figure 5: Performance of different groups $G$ for Whitening.

Theorems & Definitions (2)

• Proposition 4.1
• proof