Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustness

TL;DR

This work introduces a comprehensive benchmark for assessing Graph-LLMs robustness to adversarial attacks on graphs and their text attributes, framed around two paradigms: LLMs-as-Enhancers and LLMs-as-Predictors. It systematically evaluates structural and textual attacks using a unified pipeline across six TAG datasets, revealing that LLM-based representations generally enhance resilience compared with shallow baselines, with strong performance especially when LLMs are fine-tuned or used as predictors under certain settings. The study provides detailed analyses linking robustness to feature separability and network centrality, and releases an open-source benchmark library to enable fair, reproducible evaluations. These findings offer practical guidance for deploying Graph-LLMs in risk-sensitive applications and lay groundwork for future research in jointly challenging textual and structural perturbations.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing tasks. Recently, several LLMs-based pipelines have been developed to enhance learning on graphs with text attributes, showcasing promising performance. However, graphs are well-known to be susceptible to adversarial attacks and it remains unclear whether LLMs exhibit robustness in learning on graphs. To address this gap, our work aims to explore the potential of LLMs in the context of adversarial attacks on graphs. Specifically, we investigate the robustness against graph structural and textual perturbations in terms of two dimensions: LLMs-as-Enhancers and LLMs-as-Predictors. Through extensive experiments, we find that, compared to shallow models, both LLMs-as-Enhancers and LLMs-as-Predictors offer superior robustness against structural and textual attacks.Based on these findings, we carried out additional analyses to investigate the underlying causes. Furthermore, we have made our benchmark library openly available to facilitate quick and fair evaluations, and to encourage ongoing innovative research in this field.

Figures (14)

• Figure 1: An overview of our benchmark. The evaluation is divided into two perspectives: LLMs-as-Enhancers and LLMs-as-Predictors, both of which consider structural and textual attacks.
• Figure 2:
• Figure 3:
• Figure 4:
• Figure 5: