Reasoning-Oriented and Analogy-Based Methods for Locating and Editing in Zero-Shot Event-Relational Reasoning

TL;DR

This work tackles zero-shot event-relational reasoning by addressing the inefficiencies and opacity of prefix-tuning. It introduces ROLE to locate and edit key reasoning modules in a language model, enhancing interpretability and resource efficiency, and ABLE to transfer knowledge across tasks via analogies for improved zero-shot performance. Through experiments on ten MAVEN-derived datasets across causal and sub-event relations, ROLE shows interpretability and efficiency gains while ABLE achieves state-of-the-art results on most zero-shot tasks and provides insights into analogical transfer of editing magnitudes. Together, ROLE and ABLE offer a principled, resource-conscious framework for understanding and enhancing reasoning in large language models with practical implications for downstream event-centric NLP tasks.

Abstract

Zero-shot event-relational reasoning is an important task in natural language processing, and existing methods jointly learn a variety of event-relational prefixes and inference-form prefixes to achieve such tasks. However, training prefixes consumes large computational resources and lacks interpretability. Additionally, learning various relational and inferential knowledge inefficiently exploits the connections between tasks. Therefore, we first propose a method for Reasoning-Oriented Locating and Editing (ROLE), which locates and edits the key modules of the language model for reasoning about event relations, enhancing interpretability and also resource-efficiently optimizing the reasoning ability. Subsequently, we propose a method for Analogy-Based Locating and Editing (ABLE), which efficiently exploits the similarities and differences between tasks to optimize the zero-shot reasoning capability. Experimental results show that ROLE improves interpretability and reasoning performance with reduced computational cost. ABLE achieves SOTA results in zero-shot reasoning.

Figures (7)

• Figure 1: Comparison of knowledge transfer between existing methods and ABLE in various event-relational reasoning tasks. Top: Existing methods rely on many tasks to learn relational and reasoning knowledge, inefficiently exploiting the connections between tasks. Bottom: ABLE efficiently learns similarities and differences between tasks to enhance knowledge transfer.
• Figure 2: Overview of ROLE and ABLE. The left side shows the application of ROLE and ABLE in the Flan-T5-large encoder, and the right side shows their application in the decoder. First, ROLE is used to determine the position and editing information of tasks $A$, $B$ and $C$ (corresponding to the three vertices of the parallelogram). Then, ABLE migrates this information to task $D$ (the fourth vertex) by analogy.
• Figure 3: Heatmaps of the effect of Transformer, MLP, and Self-Attention modules on each token for each layer in the encoder, with positive samples on the left and negative samples on the right. The horizontal axis indicates the number of layers, the vertical axis indicates the tokens, and the color depth indicates the intensity of the effect. tokens are presented in 7 groups (see Appendix \ref{['appendix_C']} for details).
• Figure 4: Line plots of the effect of the Transformer, MLP, Self-Attention and Cross-Attention modules on the </s> token for each layer in the decoder, with positive samples on the left and negative samples on the right. The horizontal axis indicates the number of layers and the vertical axis indicates the intensity of the impact.
• Figure 5: Reasoning mechanism for Flan-T5-large inferring event relations.