Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference

TL;DR

This work interrogates the logical content of Natural Language Inference (NLI) by modeling the NLI relations (entailment, contradiction, neutral) as modal implications and proposing three readings: material conditional (MC), strict conditional (SC), and existential-import (EI). Using SNLI and three LLM-generated datasets, the authors construct overlapping data and inferred test sets to examine meta-inferential consistency, finding that the EI reading aligns more closely with current model predictions. The study demonstrates that modal semantics can reveal which notions of inference are captured by NLI datasets and models, and suggests that logics offer a principled framework to disentangle different inference kinds in future work. The results imply that exploiting modal readings can improve understanding of how models reason with world knowledge and language structure, with practical implications for dataset design and evaluation.

Abstract

Natural Language Inference (NLI) has been an important task for evaluating language models for Natural Language Understanding, but the logical properties of the task are poorly understood and often mischaracterized. Understanding the notion of inference captured by NLI is key to interpreting model performance on the task. In this paper we formulate three possible readings of the NLI label set and perform a comprehensive analysis of the meta-inferential properties they entail. Focusing on the SNLI dataset, we exploit (1) NLI items with shared premises and (2) items generated by LLMs to evaluate models trained on SNLI for meta-inferential consistency and derive insights into which reading of the logical relations is encoded by the dataset.