Balancing Embedding Spectrum for Recommendation

TL;DR

The paper identifies embedding collapse as a core problem in pairwise CF, where representations occupy a subspace rather than the full embedding space. It introduces DirectSpec, a spectrum-balancing approach that directly optimizes the singular-value distribution to expand the embedding spectrum, and DirectSpec+ for adaptive, individualized balancing. The authors show theoretically and empirically that balancing the spectrum can prevent collapse and improve performance, while also explaining the trade-off with alignment and connecting DirectSpec to SSL-based methods. Applied to MF and LightGCN across three datasets, DirectSpec+ yields notable gains in ranking quality with competitive efficiency, offering a practical path to more expressive recommender representations.

Abstract

Modern recommender systems heavily rely on high-quality representations learned from high-dimensional sparse data. While significant efforts have been invested in designing powerful algorithms for extracting user preferences, the factors contributing to good representations have remained relatively unexplored. In this work, we shed light on an issue in the existing pair-wise learning paradigm (i.e., the embedding collapse problem), that the representations tend to span a subspace of the whole embedding space, leading to a suboptimal solution and reducing the model capacity. Specifically, optimization on observed interactions is equivalent to a low pass filter causing users/items to have the same representations and resulting in a complete collapse. While negative sampling acts as a high pass filter to alleviate the collapse by balancing the embedding spectrum, its effectiveness is only limited to certain losses, which still leads to an incomplete collapse. To tackle this issue, we propose a novel method called DirectSpec, acting as a reliable all pass filter to balance the spectrum distribution of the embeddings during training, ensuring that users/items effectively span the entire embedding space. Additionally, we provide a thorough analysis of DirectSpec from a decorrelation perspective and propose an enhanced variant, DirectSpec+, which employs self-paced gradients to optimize irrelevant samples more effectively. Moreover, we establish a close connection between DirectSpec+ and uniformity, demonstrating that contrastive learning (CL) can alleviate the collapse issue by indirectly balancing the spectrum. Finally, we implement DirectSpec and DirectSpec+ on two popular recommender models: MF and LightGCN. Our experimental results demonstrate its effectiveness and efficiency over competitive baselines.

Figures (10)

• Figure 1: An illustration of the data distribution in the original space and expected distribution in the learning embedding space.
• Figure 2: (a) complete collapse, (b) incomplete collapse, (c) how negative sampling can balance the spectrum on Yelp: the normalized eigenvalue distribution (Top 500) of $\mathbf{A}-\mathbf{\bar{A}}$.
• Figure 3: (a): An illustration of the proposed DirectSpec, (b): A toy example running DirectSpec on a randomly generated matrix with torch.randn(10,10) $\{5, 10\}$ times.
• Figure 4: How (a): the effective rank and training loss (b): the effective rank and accuracy change with $\alpha$.
• Figure 5: The extent of collapse on three datasets.

Theorems & Definitions (8)

• Definition 1
• Definition 2
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Proposition 5
• Proposition 6