LLMTM: Benchmarking and Optimizing LLMs for Temporal Motif Analysis in Dynamic Graphs

TL;DR

The paper introduces LLMTM, a benchmark for evaluating Large Language Models on temporal motif analysis in dynamic graphs using a quadruplet edge representation and six tasks across nine motif types. It reveals that standard LLM prompting is limited by cognitive load on complex, multi-motif reasoning, while a tool-augmented LLM agent achieves high accuracy at a substantial cost. To address this, the authors propose a Structure-Aware Dispatcher that predicts problem difficulty from graph-structural and cognitive-load features and routes queries to either a lightweight LLM prompt or the expensive agent, balancing accuracy and efficiency. Extensive experiments across nine LLMs and real-world data (Enron) show strong generalization and significant cost savings from intelligent query routing, highlighting a practical path for deploying LLMs in dynamic graph analytics.

Abstract

The widespread application of Large Language Models (LLMs) has motivated a growing interest in their capacity for processing dynamic graphs. Temporal motifs, as an elementary unit and important local property of dynamic graphs which can directly reflect anomalies and unique phenomena, are essential for understanding their evolutionary dynamics and structural features. However, leveraging LLMs for temporal motif analysis on dynamic graphs remains relatively unexplored. In this paper, we systematically study LLM performance on temporal motif-related tasks. Specifically, we propose a comprehensive benchmark, LLMTM (Large Language Models in Temporal Motifs), which includes six tailored tasks across nine temporal motif types. We then conduct extensive experiments to analyze the impacts of different prompting techniques and LLMs (including nine models: openPangu-7B, the DeepSeek-R1-Distill-Qwen series, Qwen2.5-32B-Instruct, GPT-4o-mini, DeepSeek-R1, and o3) on model performance. Informed by our benchmark findings, we develop a tool-augmented LLM agent that leverages precisely engineered prompts to solve these tasks with high accuracy. Nevertheless, the high accuracy of the agent incurs a substantial cost. To address this trade-off, we propose a simple yet effective structure-aware dispatcher that considers both the dynamic graph's structural properties and the LLM's cognitive load to intelligently dispatch queries between the standard LLM prompting and the more powerful agent. Our experiments demonstrate that the structure-aware dispatcher effectively maintains high accuracy while reducing cost.

Figures (17)

• Figure 1: An overview of our LLMTM Benchmark, which includes six tasks (the question is a shortened version due to space) and nine temporal motifs. The tasks (left) are organized into two levels of increasing complexity: (1) single-temporal motif recognition and (2) multi-temporal motif identification. The nine motifs are illustrated on the right.
• Figure 2: An overview of our framework for balancing the accuracy-cost trade-off. A "Structure-Aware Dispatcher" first extracts the dynamic graph from natural language, predicts the query's difficulty, and then strategically routes simple queries to a standard LLM (for low cost); complex ones to our tool-augmented agent. The agent follows a workflow of task planning, tool selection, tool calling, and response generation to achieve high accuracy, albeit at a greater computational cost.
• Figure 3: Performance of the tool-augmented Agent versus GPT-4o-mini on the "Motif Detection" task, comparing (a) accuracy and (b) average token consumption. See Appendix \ref{['app:radas_graph']} for results on all other tasks.
• Figure 4: An example of a Chain-of-Thought (CoT) prompt for the “Motif Detection” task.
• Figure 5: The prompt template for the Tool-Augmented LLM Agent.