LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction

TL;DR

The paper tackles document-level event argument extraction (EAE) under limited labeled data by exploiting in-context learning (ICL) in large language models. It introduces HD-LoA prompting, which combines explicit task heuristics (heuristic-driven demonstration construction) with link-of-analogy prompting to handle many event types and unseen roles. Empirical results on RAMS and DocEE show consistent gains over standard and chain-of-thought prompting, with strong few-shot performance and cross-domain robustness, plus transferable gains to sentiment analysis and natural language inference. The work provides evidence that LLMs can learn task-specific heuristics from demonstrations and offers a practical prompting strategy to reduce data requirements while maintaining high accuracy.

Abstract

In this study, we investigate in-context learning (ICL) in document-level event argument extraction (EAE) to alleviate the dependency on large-scale labeled data for this task. We introduce the Heuristic-Driven Link-of-Analogy (HD-LoA) prompting to address the challenge of example selection and to develop a prompting strategy tailored for EAE. Specifically, we hypothesize and validate that LLMs learn task-specific heuristics from demonstrations via ICL. Building upon this hypothesis, we introduce an explicit heuristic-driven demonstration construction approach, which transforms the haphazard example selection process into a methodical method that emphasizes task heuristics. Additionally, inspired by the analogical reasoning of human, we propose the link-of-analogy prompting, which enables LLMs to process new situations by drawing analogies to known situations, enhancing their performance on unseen classes beyond limited ICL examples. Experiments show that our method outperforms existing prompting methods and few-shot supervised learning methods on document-level EAE datasets. Additionally, the HD-LoA prompting shows effectiveness in diverse tasks like sentiment analysis and natural language inference, demonstrating its broad adaptability.

Figures (7)

• Figure 1: CoT's step-by-step reasoning degrades to a single step for non-reasoning tasks. Reasoning steps of reasoning tasks (in orange) and non-reasoning tasks (in blue) are compared. Different colors indicate distinct reasoning steps. Prompts are from shum-etal-2023-automatic.
• Figure 2: Heuristics are implicitly embedded within explanations of in-context examples.
• Figure 3: An illustration of the correlation between example quantity and heuristic diversity in well-designed prompts. # Examples: the number of examples used in each prompt of the corresponding paper. # Heuristics: the number of heuristics identified in each prompt of the corresponding paper. # Heuristics in Rand.: the average number of heuristics in the randomly constructed prompt.
• Figure 4: Comparison of ICL performance using single-heuristic strategy versus diverse-heuristics strategy across different number of example on the StrategyQA and SST-2 Dataset.
• Figure 5: An illustration of HD-LoA prompting.