A Comprehensive Evaluation on Event Reasoning of Large Language Models

TL;DR

This work introduces EV^{2}, a two-level benchmark for contextualized event reasoning that jointly probes event schema knowledge (schema level) and event reasoning abilities (instance level) across multiple relation types (Causes, IsResult, Before, After, IsSubevent, HasSubevent) and reasoning paradigms (CEC and CRR). By combining automatic graph synthesis with human curation and annotation, EV^{2} enables rigorous evaluation of both knowledge and reasoning, revealing that current LLMs exhibit nontrivial event reasoning but remain far from satisfactory, with notable imbalances across relations and paradigms. The study finds that LLMs possess event schema knowledge to some extent but struggle to align its use with human expectations, and that explicit guidance leveraging schema knowledge can substantially improve reasoning performance, suggesting a memory-based approach to integrate event schemas into LLM systems. Overall, EV^{2} provides a comprehensive framework for diagnosing and improving event reasoning in LLMs, with practical implications for memory-augmented reasoning and domain-specific AI agents.

Abstract

Event reasoning is a fundamental ability that underlies many applications. It requires event schema knowledge to perform global reasoning and needs to deal with the diversity of the inter-event relations and the reasoning paradigms. How well LLMs accomplish event reasoning on various relations and reasoning paradigms remains unknown. To mitigate this disparity, we comprehensively evaluate the abilities of event reasoning of LLMs. We introduce a novel benchmark EV2 for EValuation of EVent reasoning. EV2 consists of two levels of evaluation of schema and instance and is comprehensive in relations and reasoning paradigms. We conduct extensive experiments on EV2. We find that LLMs have abilities to accomplish event reasoning but their performances are far from satisfactory. We also notice the imbalance of event reasoning abilities in LLMs. Besides, LLMs have event schema knowledge, however, they're not aligned with humans on how to utilize the knowledge. Based on these findings, we guide the LLMs in utilizing the event schema knowledge as memory leading to improvements on event reasoning.

Figures (5)

• Figure 1: An example of event reasoning. The red words are event schema knowledge. The sentences below are event instances. In event reasoning, there are various paradigms such as Contextual Event Classification (CEC) and Contextual Relation Reasoning (CRR), and diverse inter-event relations.
• Figure 2: Results of CEC and CRR. S and I stand for schema- and instance-level. Relation types of Causality, Temporality, and Hierarchy are denoted as C, T, and H.
• Figure 3: Improvements trend on CEC and CRR. The dashline represents the balanced improvement with slope 3/4 considering the CEC is a 4-way multiple-choice task while CRR has three choices. The red line is the regression line of models except GPT4.
• Figure 4: Comparisons between performances on instance- and schema-level. The dashed line represents the balanced improvement with slope 1. The red line is the regression line of all models
• Figure 5: Annotation platform. This Figure shows the process of annotating one data.