Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entailed Between the Lines: Incorporating Implication into NLI

Shreya Havaldar, Hamidreza Alvari, John Palowitch, Mohammad Javad Hosseini, Senaka Buthpitiya, Alex Fabrikant

TL;DR

This paper argues that human communication relies heavily on implicit meaning that standard NLI datasets fail to capture. It formalizes implied entailment by distinguishing explicit and implicit entailment, and introduces the Implied NLI (INLI) dataset (10k premises, 40k hypotheses) generated via a two-stage pipeline that augments implicature frames and creates alternative hypotheses, followed by thorough human validation. Empirical results show current NLI benchmarks underrepresent implied entailments, and models trained on INLI better recognize implied entailments, with strong generalization across domains and datasets. The work demonstrates that fine-tuning on INLI improves models' ability to read between the lines, with implications for real-world tasks like translation, summarization, and content evaluation.

Abstract

Much of human communication depends on implication, conveying meaning beyond literal words to express a wider range of thoughts, intentions, and feelings. For models to better understand and facilitate human communication, they must be responsive to the text's implicit meaning. We focus on Natural Language Inference (NLI), a core tool for many language tasks, and find that state-of-the-art NLI models and datasets struggle to recognize a range of cases where entailment is implied, rather than explicit from the text. We formalize implied entailment as an extension of the NLI task and introduce the Implied NLI dataset (INLI) to help today's LLMs both recognize a broader variety of implied entailments and to distinguish between implicit and explicit entailment. We show how LLMs fine-tuned on INLI understand implied entailment and can generalize this understanding across datasets and domains.

Entailed Between the Lines: Incorporating Implication into NLI

TL;DR

This paper argues that human communication relies heavily on implicit meaning that standard NLI datasets fail to capture. It formalizes implied entailment by distinguishing explicit and implicit entailment, and introduces the Implied NLI (INLI) dataset (10k premises, 40k hypotheses) generated via a two-stage pipeline that augments implicature frames and creates alternative hypotheses, followed by thorough human validation. Empirical results show current NLI benchmarks underrepresent implied entailments, and models trained on INLI better recognize implied entailments, with strong generalization across domains and datasets. The work demonstrates that fine-tuning on INLI improves models' ability to read between the lines, with implications for real-world tasks like translation, summarization, and content evaluation.

Abstract

Much of human communication depends on implication, conveying meaning beyond literal words to express a wider range of thoughts, intentions, and feelings. For models to better understand and facilitate human communication, they must be responsive to the text's implicit meaning. We focus on Natural Language Inference (NLI), a core tool for many language tasks, and find that state-of-the-art NLI models and datasets struggle to recognize a range of cases where entailment is implied, rather than explicit from the text. We formalize implied entailment as an extension of the NLI task and introduce the Implied NLI dataset (INLI) to help today's LLMs both recognize a broader variety of implied entailments and to distinguish between implicit and explicit entailment. We show how LLMs fine-tuned on INLI understand implied entailment and can generalize this understanding across datasets and domains.
Paper Structure (49 sections, 4 figures, 12 tables)

This paper contains 49 sections, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Formalizing Implied Entailment
  3. Implied Language Understanding in Current Benchmarks & Models
  4. NLI Benchmarks Contain Few Implied Entailments
  5. Human validation of Table \ref{['tab:curr-benchmarks']}.
  6. NLI Models Rarely Generalize to Implied Entailments
  7. The Implied NLI Dataset (INLI)
  8. Augmenting Implicature Frames
  9. Conversational implicatures.
  10. Situational implicatures.
  11. Alternative Hypothesis Generation
  12. Paraphrasing.
  13. Data robustness.
  14. Human Validation of Dataset
  15. Training human annotators.
  16. ...and 34 more sections

Figures (4)

  • Figure 1: Extending NLI to introduce implication. We propose that entailment models should treat implications as entailed, while also distinguishing between explicit and implicit entailments.
  • Figure 2: Stage 1: Implicature Augmentation. Given existing data that contains an implicature (e.g. the answer to an indirect question, a widely-accepted social norm, etc.), we augment the data into a premise and implicitly entailed hypothesis. In this figure we show augmented examples from Ludwig and SocialChem.
  • Figure 3: Stage 2: Alternative Hypothesis Generation. Given premises and their corresponding implied entailments, we generate three additional hypotheses (explicit, neutral, and contradictory) by subtly modifying the implied entailment. This ensures we build a more challenging dataset for the implied entailment task.
  • Figure 4: Defining implied entailment. We propose implied entailments should be evaluated via two axes -- inferability and implicitness.