SE-GCL: An Event-Based Simple and Effective Graph Contrastive Learning for Text Representation

TL;DR

SE-GCL introduces an event-based graph contrastive learning framework for text representation that prioritizes core semantic information through intra-relation graphs and event skeletons. It constructs lightweight embeddings via an MLP for anchors, shuffles for negatives, and derives two positive embeddings from structure and event information, optimized with a multi-loss regime that enforces inter-class separation while controlling intra-class variation. Experiments on AG News, 20NG, SougouNews, and THUCNews show SE-GCL achieving strong performance and efficiency, with ablations confirming the importance of each component. The approach offers a scalable, unsupervised alternative to traditional augmentation-heavy GCL methods and holds promise for longer, event-rich texts and potential multi-label extensions.

Abstract

Text representation learning is significant as the cornerstone of natural language processing. In recent years, graph contrastive learning (GCL) has been widely used in text representation learning due to its ability to represent and capture complex text information in a self-supervised setting. However, current mainstream graph contrastive learning methods often require the incorporation of domain knowledge or cumbersome computations to guide the data augmentation process, which significantly limits the application efficiency and scope of GCL. Additionally, many methods learn text representations only by constructing word-document relationships, which overlooks the rich contextual semantic information in the text. To address these issues and exploit representative textual semantics, we present an event-based, simple, and effective graph contrastive learning (SE-GCL) for text representation. Precisely, we extract event blocks from text and construct internal relation graphs to represent inter-semantic interconnections, which can ensure that the most critical semantic information is preserved. Then, we devise a streamlined, unsupervised graph contrastive learning framework to leverage the complementary nature of the event semantic and structural information for intricate feature data capture. In particular, we introduce the concept of an event skeleton for core representation semantics and simplify the typically complex data augmentation techniques found in existing graph contrastive learning to boost algorithmic efficiency. We employ multiple loss functions to prompt diverse embeddings to converge or diverge within a confined distance in the vector space, ultimately achieving a harmonious equilibrium. We conducted experiments on the proposed SE-GCL on four standard data sets (AG News, 20NG, SougouNews, and THUCNews) to verify its effectiveness in text representation learning.

Figures (6)

• Figure 1: Different processing methods for text representation.
• Figure 2: Illustration of our method SE-GCL. First, SE-GCL builds intra-relation graphs for texts and identifies their event skeletons from them. An MLP is used to generate the word node anchor embedding $H^+$ of intra-relation graphs, and the negative embedding $H^-$ is obtained by perturbing the anchor embedding, and the event embedding $H^+_e$ is obtained by sampling. In addition, a GCN will be used to generate structural embeddings $H^+_s$. A contrastive loss is then applied to close the distance between positive embeddings and anchor embeddings while widening the distance between negative embeddings and anchor embeddings.
• Figure 3: The overall process of constructing intra-relation graph.
• Figure 4: Experiment results in different settings ($\theta$ and $\eta$).
• Figure 5: Experiment results in different settings ($W_e$ and $W_s$)