Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification Using Graph Neural Networks?
Margarita Bugueño, Gerard de Melo
TL;DR
This paper provides a comprehensive, cross-domain evaluation of graph-based text representations for classification using Graph Neural Networks. By systematically comparing intuitive (co-occurrence and sequence) graphs and TextLevelGCN against BoW and Transformer baselines across five datasets, it disentangles the impact of graph construction from message passing and node initialization. Key findings show that simple graph constructions are particularly effective for longer documents, often outperforming Transformer-based models in those settings, and that graph methods can be substantially more efficient. The work also reveals that pre-trained static embeddings can outperform contextualized embeddings for certain graph setups, underscoring practical guidance for deploying graph-based text classifiers. The authors provide code and discuss open challenges, including learning graph structure end-to-end to reduce design dependence and domain sensitivity.
Abstract
Given the success of Graph Neural Networks (GNNs) for structure-aware machine learning, many studies have explored their use for text classification, but mostly in specific domains with limited data characteristics. Moreover, some strategies prior to GNNs relied on graph mining and classical machine learning, making it difficult to assess their effectiveness in modern settings. This work extensively investigates graph representation methods for text classification, identifying practical implications and open challenges. We compare different graph construction schemes using a variety of GNN architectures and setups across five datasets, encompassing short and long documents as well as unbalanced scenarios in diverse domains. Two Transformer-based large language models are also included to complement the study. The results show that i) although the effectiveness of graphs depends on the textual input features and domain, simple graph constructions perform better the longer the documents are, ii) graph representations are especially beneficial for longer documents, outperforming Transformer-based models, iii) graph methods are particularly efficient at solving the task.