LLM-based Few-Shot Early Rumor Detection with Imitation Agent

TL;DR

The paper tackles early rumor detection under data scarcity by proposing a decoupled framework that uses a lightweight autonomous agent to determine when to invoke a powerful LLM detector. EARD is formulated as an MDP and learned via imitation learning using three expert trajectories (Conservative, Early-Action, Misleading) to optimize timing while avoiding false starts. Across four real-world datasets and multiple LLMs, the approach improves macro-F1 and reduces Early Rate compared to existing EARD methods and LLM-only baselines, demonstrating robust generalization to unseen events and resilience to limited supervision. The framework achieves cost-efficient, timely, and accurate rumor detection by minimizing LLM inferences to a single, strategically-timed prediction.

Abstract

Early Rumor Detection (EARD) aims to identify the earliest point at which a claim can be accurately classified based on a sequence of social media posts. This is especially challenging in data-scarce settings. While Large Language Models (LLMs) perform well in few-shot NLP tasks, they are not well-suited for time-series data and are computationally expensive for both training and inference. In this work, we propose a novel EARD framework that combines an autonomous agent and an LLM-based detection model, where the agent acts as a reliable decision-maker for \textit{early time point determination}, while the LLM serves as a powerful \textit{rumor detector}. This approach offers the first solution for few-shot EARD, necessitating only the training of a lightweight agent and allowing the LLM to remain training-free. Extensive experiments on four real-world datasets show our approach boosts performance across LLMs and surpasses existing EARD methods in accuracy and earliness.

Figures (6)

• Figure 1: llustration of the proposed EARD framework. During training, three types of expert trajectories are generated based on the LLM's predictions and the label. We utilize imitation learning to train the agent to find an optimal policy $\pi$ that aligns itself with CE and EAE while moves away from ME. At Inference, the trained agent automatically determines an early time point for the LLM to perform rumor detection. CE: Conservative Expert; EAE: Early-Action Expert; ME: Misleading Expert.
• Figure 2: An intuitive illustration of probability of stop action for CE, EAE, and ME over time. When the agent aligns itself with EAE and CE, it may hover around $0.5$ at some time points (e.g., $t=6$), leading to random choices. By including ME, the agent is encouraged to move away from the state-action distribution of ME, thereby increasing the stop probability.
• Figure 3: Comparison of ChatGPT (with Preset Time Checkpoints Strategy) and our method. The x-axis indicates different time intervals of posts used for prediction. The top row displays macro-F1 scores, while the bottom row shows the ER.
• Figure 4: Results of base LLMs evaluated directly and the LLMs incorporated with our agent trained on PHEME and TWITTER, which are tested on Twitter-COVID-19. Bars with diagonal hatching ("/") indicate models trained directly on Twitter-COVID-19.
• Figure 5: Trade-off between classification accuracy and earliness for our method (across different expert trajectory ratios) and existing EARD methods. Each marker shows average performance over 4 datasets. Our settings are circles, color-coded by $\alpha$ (CE). Gold outlines mark Pareto-optimal points, connected by the dashed black line to form the Pareto frontier.

Theorems & Definitions (4)

• proposition 1
• proposition 2
• proposition 3
• proposition 4