Unleashing the Power of Emojis in Texts via Self-supervised Graph Pre-Training

TL;DR

A graph pre-train framework for text and emoji co-modeling, which contains two graph pre-training tasks: node-level graph contrastive learning and edge-level link reconstruction learning, which proves significant improvement over previous strong baseline methods.

Abstract

Emojis have gained immense popularity on social platforms, serving as a common means to supplement or replace text. However, existing data mining approaches generally either completely ignore or simply treat emojis as ordinary Unicode characters, which may limit the model's ability to grasp the rich semantic information in emojis and the interaction between emojis and texts. Thus, it is necessary to release the emoji's power in social media data mining. To this end, we first construct a heterogeneous graph consisting of three types of nodes, i.e. post, word and emoji nodes to improve the representation of different elements in posts. The edges are also well-defined to model how these three elements interact with each other. To facilitate the sharing of information among post, word and emoji nodes, we propose a graph pre-train framework for text and emoji co-modeling, which contains two graph pre-training tasks: node-level graph contrastive learning and edge-level link reconstruction learning. Extensive experiments on the Xiaohongshu and Twitter datasets with two types of downstream tasks demonstrate that our approach proves significant improvement over previous strong baseline methods.

Figures (2)

• Figure 1: An illustration of all processes of our model. Part a) shows the construction of our heterogeneous graph that incorporates post, emoji and word nodes, along with their connections. Part b) shows the structure of node level and edge level pre-training tasks. Part c) shows the structure of two downstream tasks.
• Figure 2: The t-SNE visualization results of emoji embeddings from four models.