Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAFT: Structure-aware Transformers for Textual Interaction Classification

Hongtao Wang, Renchi Yang, Hewen Wang, Haoran Zheng, Jianliang Xu

TL;DR

This work tackles Textual Interaction Classification on bipartite Textual Interaction Networks by marrying pretrained language models with structure-aware transformers. SAFT introduces line graph attention and gated attention units, a proxy token to couple token- and edge-level signals, and fast, SVD-based distance and centrality embeddings to capture local and global topology, complemented by graph sampling for scalability. Empirical results across eight real-world datasets show SAFT consistently outperforming a wide range of baselines, with ablations confirming the contributions of MP, distance embeddings, centrality embeddings, and sampling. The approach offers a practical, scalable framework for TIC that leverages both deep textual semantics and topology-aware features, with broad implications for spam detection, fraud identification, and sentiment analysis in complex networks.

Abstract

Textual interaction networks (TINs) are an omnipresent data structure used to model the interplay between users and items on e-commerce websites, social networks, etc., where each interaction is associated with a text description. Classifying such textual interactions (TIC) finds extensive use in detecting spam reviews in e-commerce, fraudulent transactions in finance, and so on. Existing TIC solutions either (i) fail to capture the rich text semantics due to the use of context-free text embeddings, and/or (ii) disregard the bipartite structure and node heterogeneity of TINs, leading to compromised TIC performance. In this work, we propose SAFT, a new architecture that integrates language- and graph-based modules for the effective fusion of textual and structural semantics in the representation learning of interactions. In particular, line graph attention (LGA)/gated attention units (GAUs) and pretrained language models (PLMs) are capitalized on to model the interaction-level and token-level signals, which are further coupled via the proxy token in an iterative and contextualized fashion. Additionally, an efficient and theoretically-grounded approach is developed to encode the local and global topology information pertaining to interactions into structural embeddings. The resulting embeddings not only inject the structural features underlying TINs into the textual interaction encoding but also facilitate the design of graph sampling strategies. Extensive empirical evaluations on multiple real TIN datasets demonstrate the superiority of SAFT over the state-of-the-art baselines in TIC accuracy.

SAFT: Structure-aware Transformers for Textual Interaction Classification

TL;DR

This work tackles Textual Interaction Classification on bipartite Textual Interaction Networks by marrying pretrained language models with structure-aware transformers. SAFT introduces line graph attention and gated attention units, a proxy token to couple token- and edge-level signals, and fast, SVD-based distance and centrality embeddings to capture local and global topology, complemented by graph sampling for scalability. Empirical results across eight real-world datasets show SAFT consistently outperforming a wide range of baselines, with ablations confirming the contributions of MP, distance embeddings, centrality embeddings, and sampling. The approach offers a practical, scalable framework for TIC that leverages both deep textual semantics and topology-aware features, with broad implications for spam detection, fraud identification, and sentiment analysis in complex networks.

Abstract

Textual interaction networks (TINs) are an omnipresent data structure used to model the interplay between users and items on e-commerce websites, social networks, etc., where each interaction is associated with a text description. Classifying such textual interactions (TIC) finds extensive use in detecting spam reviews in e-commerce, fraudulent transactions in finance, and so on. Existing TIC solutions either (i) fail to capture the rich text semantics due to the use of context-free text embeddings, and/or (ii) disregard the bipartite structure and node heterogeneity of TINs, leading to compromised TIC performance. In this work, we propose SAFT, a new architecture that integrates language- and graph-based modules for the effective fusion of textual and structural semantics in the representation learning of interactions. In particular, line graph attention (LGA)/gated attention units (GAUs) and pretrained language models (PLMs) are capitalized on to model the interaction-level and token-level signals, which are further coupled via the proxy token in an iterative and contextualized fashion. Additionally, an efficient and theoretically-grounded approach is developed to encode the local and global topology information pertaining to interactions into structural embeddings. The resulting embeddings not only inject the structural features underlying TINs into the textual interaction encoding but also facilitate the design of graph sampling strategies. Extensive empirical evaluations on multiple real TIN datasets demonstrate the superiority of SAFT over the state-of-the-art baselines in TIC accuracy.

Paper Structure

This paper contains 39 sections, 4 theorems, 25 equations, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semi-Supervised Edge Classification
  4. Textual Graph Representation Learning
  5. Preliminaries
  6. Notations and Terminology
  7. Graph Neural Networks (GNNs)
  8. Transformer
  9. Methodology
  10. Textual Interaction Encoding
  11. Token Initialization
  12. Embedding Updating
  13. Final Representations
  14. User-wise and Item-wise Message Passing
  15. LGA-based Method
  16. ...and 24 more sections

Key Result

lemma 1

${\boldsymbol{P}^{(\textrm{u})}}\xspace=\textsf{ssoftmax}\left({{\boldsymbol{E}^{(\textrm{u})}}\xspace} {{\boldsymbol{E}^{(\textrm{u})}}\xspace}^{\top}/\sqrt{|\mathcal{U}\xspace|}\right)$ and ${\boldsymbol{P}^{(\textrm{i})}}\xspace=\textsf{ssoftmax}\left({{\boldsymbol{E}^{(\textrm{i})}}\xspace} {{\b

Figures (8)

  • Figure 1: The Overall Architecture of Our Proposed SAFT.
  • Figure 2: Local and Global Structural Patterns in TIN $\mathcal{G}\xspace$.
  • Figure 3: SAFT with various graph sampling strategies
  • Figure 4: Training time of SAFT and baselines.
  • Figure 5: Macro-F1 and Micro-F1 when varying #layers
  • ...and 3 more figures

Theorems & Definitions (5)

  • definition 1: Textual Interaction Classification
  • lemma 1
  • theorem 1
  • theorem 2
  • theorem 3