A Logical Pattern Memory Pre-trained Model for Entailment Tree Generation

TL;DR

The logical pattern memory pre-trained model (LMPM) incorporates an external memory structure to learn and store the latent representations of logical patterns, which aids in generating logically consistent conclusions.

Abstract

Generating coherent and credible explanations remains a significant challenge in the field of AI. In recent years, researchers have delved into the utilization of entailment trees to depict explanations, which exhibit a reasoning process of how a hypothesis is deduced from the supporting facts. However, existing models often overlook the importance of generating intermediate conclusions with logical consistency from the given facts, leading to inaccurate conclusions and undermining the overall credibility of entailment trees. To address this limitation, we propose the logical pattern memory pre-trained model (LMPM). LMPM incorporates an external memory structure to learn and store the latent representations of logical patterns, which aids in generating logically consistent conclusions. Furthermore, to mitigate the influence of logically irrelevant domain knowledge in the Wikipedia-based data, we introduce an entity abstraction approach to construct the dataset for pre-training LMPM. The experimental results highlight the effectiveness of our approach in improving the quality of entailment tree generation. By leveraging logical entailment patterns, our model produces more coherent and reasonable conclusions that closely align with the underlying premises. Code and Data are released at https://github.com/YuanLi95/T5-LMPM

Figures (9)

• Figure 1: Illustration of Entailment Tree Generation. The top half presents the inputs, while the generated entailment tree is depicted in the bottom half. The tree consists of a hypothesis (pink), premises (blue), and generated intermediate conclusions (green).
• Figure 2: Examples of Logical Patterns. Each box represents a single-step deductive process. The texts within the green and blue boxes are extracted from the Wikipedia-based synthetic dataset and the entailment tree corpus, respectively. The text within the gray box is the logical patterns obtained through entity abstraction, where <E> denotes the special token <extra_id_ > reserved in T5.
• Figure 3: The overall architecture of LMPM
• Figure 4: Examples of complex deductions. The gray boxes represent two distinct logical patterns that need to be combined to generate the appropriate conclusion shown in the blue box.
• Figure 5: The impact of logical pattern pre-training data size on performance. The AllCorrect scores for Overall and Inter (Intermediates) are provided.