IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach

TL;DR

This work tackles causal relation identification in news text under CASE-2022 by framing the task as masked language modeling in a few-shot, prompt-based setup. It demonstrates that a model fine-tuned with as few as $k$ per class (e.g., $k=256$) and augmented prompts can achieve competitive precision, accuracy, and F1 compared to ensemble methods trained on all available data, reducing data requirements to about $15.7\%$ and enabling robust model selection. Compared with ensemble baselines, the prompt-based approach generalizes well to the official test set, achieving strong scores (e.g., a near-top F1 around $0.86$) while maintaining higher data-efficiency. The study also analyzes dataset properties and ensemble strategies, highlighting the practical value of prompt-based few-shot methods for CRI in low-resource settings and rapid evaluation. Overall, the work suggests that prompt-based MLM with demonstrations offers a viable, efficient alternative for causal event classification in NLP applications.

Abstract

In this paper, we describe our participation in the subtask 1 of CASE-2022, Event Causality Identification with Casual News Corpus. We address the Causal Relation Identification (CRI) task by exploiting a set of simple yet complementary techniques for fine-tuning language models (LMs) on a small number of annotated examples (i.e., a few-shot configuration). We follow a prompt-based prediction approach for fine-tuning LMs in which the CRI task is treated as a masked language modeling problem (MLM). This approach allows LMs natively pre-trained on MLM problems to directly generate textual responses to CRI-specific prompts. We compare the performance of this method against ensemble techniques trained on the entire dataset. Our best-performing submission was fine-tuned with only 256 instances per class, 15.7% of the all available data, and yet obtained the second-best precision (0.82), third-best accuracy (0.82), and an F1-score (0.85) very close to what was reported by the winner team (0.86).

Figures (2)

• Figure 1: Augmented prompt-based classification for causality identification task. First, the input instance $x = \textit{"Soldiers were hurt in the attacks"}$ is converted into three different input prompts by applying $f_t'(x)$ three times. Then, these three prompts are given to a RoBERTa model, and one logit vector is obtained for each. These vectors are then averaged, and the word with the highest score, "causal", is selected. Finally, this word is mapped to its corresponding class, and $x$ is classified as positive. Note that, in this example, we have the following word-to-class label mapping $word(positive) = \textit{"causal"}$ and $word(negative) = \textit{"random"}$.
• Figure 2: Our proposed method to ensemble $N$ fine-tuned LMs, based on fajcik2019but approach. We fine-tune several LMs by modifying only the training seed. Our implementation uses the sequence classification task from HuggingFace toolkit wolf2020transformerslhoest2021datasets.