5W1H Extraction With Large Language Models

TL;DR

The paper tackles 5W1H extraction from news using large language models by creating a high-quality, multi-domain dataset (CNN/DailyMail, XSum, NYT, RA-MDS) and evaluating both zero-shot/few-shot prompting and efficient fine-tuning with QLoRA on models such as LLaMa, Vicuna, and Guanaco. It demonstrates that carefully fine-tuned 13B models outperform ChatGPT on 5W1H extraction, while GPT-4 can approach fine-tuned performance with few-shot prompts; zero-shot ChatGPT struggles. The work also explores cross-domain transfer, showing robust extraction for concise elements (Who/Where/When) but more challenging comprehensive ones (What/Why/How), and argues that a high-quality annotated dataset is crucial for enabling reliable cross-domain 5W1H extraction. Overall, the study provides a practical pipeline for efficient, domain-adaptive 5W1H extraction in news using resource-aware fine-tuning and rigorous evaluation.

Abstract

The extraction of essential news elements through the 5W1H framework (\textit{What}, \textit{When}, \textit{Where}, \textit{Why}, \textit{Who}, and \textit{How}) is critical for event extraction and text summarization. The advent of Large language models (LLMs) such as ChatGPT presents an opportunity to address language-related tasks through simple prompts without fine-tuning models with much time. While ChatGPT has encountered challenges in processing longer news texts and analyzing specific attributes in context, especially answering questions about \textit{What}, \textit{Why}, and \textit{How}. The effectiveness of extraction tasks is notably dependent on high-quality human-annotated datasets. However, the absence of such datasets for the 5W1H extraction increases the difficulty of fine-tuning strategies based on open-source LLMs. To address these limitations, first, we annotate a high-quality 5W1H dataset based on four typical news corpora (\textit{CNN/DailyMail}, \textit{XSum}, \textit{NYT}, \textit{RA-MDS}); second, we design several strategies from zero-shot/few-shot prompting to efficient fine-tuning to conduct 5W1H aspects extraction from the original news documents. The experimental results demonstrate that the performance of the fine-tuned models on our labelled dataset is superior to the performance of ChatGPT. Furthermore, we also explore the domain adaptation capability by testing the source-domain (e.g. NYT) models on the target domain corpus (e.g. CNN/DailyMail) for the task of 5W1H extraction.

Figures (4)

• Figure 1: Different colors represent different sources of information from the original text.In the context of the article, the mention of Hillary is not meant to be a specific name, but the answer provided by ChatGPT mistakenly identifies it as one of the important people. The names of places mentioned in the green line may also refer to residential areas, but the given answers do not correspond to them.
• Figure 2: Fine-tuning LLMs on 5W1H elements extraction from news articles. Construct annotated datasets, convert them into the required format for input into LLMs, fine-tune with different models, and select the best model based on evaluation results.
• Figure 3: Correspondence between 5W1H aspects and the original text, with different colors representing different answers to questions.
• Figure 4: Effective numbers of different models answering 5W1H questions on CNN/DailyMail dataset. All models are based on 13B size. Different colors represent different questions.