Modeling Sequences as Star Graphs to Address Over-smoothing in Self-attentive Sequential Recommendation

TL;DR

The analysis of the attention weights learned in SA-based methods indicates that on sparse recommendation data, modeling dependencies in all item pairs using the SA mechanism yields limited information gain, and thus, might not benefit the recommendation performance.

Abstract

Self-attention (SA) mechanisms have been widely used in developing sequential recommendation (SR) methods, and demonstrated state-of-the-art performance. However, in this paper, we show that self-attentive SR methods substantially suffer from the over-smoothing issue that item embeddings within a sequence become increasingly similar across attention blocks. As widely demonstrated in the literature, this issue could lead to a loss of information in individual items, and significantly degrade models' scalability and performance. To address the over-smoothing issue, in this paper, we view items within a sequence constituting a star graph and develop a method, denoted as MSSG, for SR. Different from existing self-attentive methods, MSSG introduces an additional internal node to specifically capture the global information within the sequence, and does not require information propagation among items. This design fundamentally addresses the over-smoothing issue and enables MSSG a linear time complexity with respect to the sequence length. We compare MSSG with ten state-of-the-art baseline methods on six public benchmark datasets. Our experimental results demonstrate that MSSG significantly outperforms the baseline methods, with an improvement of as much as 10.10%. Our analysis shows the superior scalability of MSSG over the state-of-the-art self-attentive methods. Our complexity analysis and run-time performance comparison together show that MSSG is both theoretically and practically more efficient than self-attentive methods. Our analysis of the attention weights learned in SA-based methods indicates that on sparse recommendation data, modeling dependencies in all item pairs using the SA mechanism yields limited information gain, and thus, might not benefit the recommendation performance

Figures (9)

• Figure 1: Illustration of a sequence represented by different graphs, in which $v_{s_t}$ is the $t$-th item in the sequence and $c$ is the internal node in the star graph.
• Figure 2: The overall architecture of $\mathop{\mathtt{MSSG}}\limits$. $\mathop{\mathtt{MSSG}}\limits$ models sequences using star graphs and utilizes the internal node $c$ to integrate information from all item nodes.
• Figure 3: Performance on users of different activity levels
• Figure 4: The average similarity $a^{m}$ in different blocks of $\mathop{\mathtt{MSSG}}\limits$ and $\mathop{\mathtt{SASRec}}\limits$
• Figure 5: Performance over different numbers of blocks