EVIT: Event-Oriented Instruction Tuning for Event Reasoning

TL;DR

EvIT addresses the deficiency of small instruction-tuned LLMs in event reasoning by introducing an explicit event-centric representation and learning objective. It defines a novel event quadruple ${\mathcal{Q}} = ({\mathcal{C}}, {\mathcal{E}}^{h}, {\mathcal{R}}, {\mathcal{E}}^{t})$ and trains via generation and discrimination within an instruction-tuning framework, encapsulated as templates derived with ChatGPT. The method constructs a large, diversified EvIT dataset from BookCorpus and related resources, and finetunes a Llama-7B model to form EvIT. Experiments on eight event-reasoning tasks show EvIT surpasses strong baselines in automatic and human evaluations for both close and open settings, demonstrating improved event understanding, relations, and prediction capability. The work advances event-centric training for LLMs and suggests future multimodal extensions to broaden event reasoning beyond text.

Abstract

Events refer to specific occurrences, incidents, or happenings that take place under a particular background. Event reasoning aims to infer events according to certain relations and predict future events. The cutting-edge techniques for event reasoning play a crucial role in various natural language processing applications. Large language models (LLMs) have made significant advancements in event reasoning owing to their wealth of knowledge and reasoning capabilities. However, smaller instruction-tuned models currently in use do not consistently demonstrate exceptional proficiency in managing these tasks. This discrepancy arises from the absence of explicit modeling of events and the interconnections of them within their instruction data. Consequently, these models face challenges in comprehending event structures and semantics while struggling to bridge the gap between their interpretations and human understanding of events. Additionally, their limitations in grasping event relations lead to constrained event reasoning abilities to effectively deduce and incorporate pertinent event knowledge. In this paper, we propose Event-Oriented Instruction Tuning (EvIT) to train our LLM. Specifically, we first propose a novel structure named event quadruple which contains the structure and semantics of events and is complete in the event representation. We then design event-relation learning based on the structures. We encapsulate the learning into the instruction-tuning formulation to better stimulate the event reasoning capacity of our model. We design a heuristic unsupervised method to mine event quadruple from a large-scale corpus. At last, we finetune a Llama model on our Event-Oriented Instruction Tuning. We conduct extensive experiments on event reasoning tasks on several datasets. Automatic and human evaluations demonstrate EvIT achieves competitive performances on event reasoning.

Figures (7)

• Figure 1: Overview of training process and evaluation of $\textsc{EvIT}$. The training process encompasses Event-Oriented Instruction Tuning and Construction of event quadruple.
• Figure 2: (a) ChatGPT input prompt of Before relation of discrimination learning with context. (b) The ChatGPT generation examples of query in (a). [event] and [context] are placeholders for the head event ${\mathcal{E}}^{h}$ and context ${\mathcal{C}}$. (c) Template for encapsulating event candidates. (d) The final input for our event-relation training.
• Figure 3: Wordcloud of verbs of events.
• Figure 4: Statistic of the length of events.
• Figure 5: Evaluation prompts for all models.