LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference

TL;DR

LiTEx presents a linguistically informed taxonomy to capture within-label variation in NLI explanations, distinguishing TB and WK reasoning across eight categories. The taxonomy is validated with high inter-annotator agreement and learnable automatic classification, and it enhances explanation generation: taxonomy-guided prompts yield explanations more aligned with human reasoning than highlights or baseline prompts. The study demonstrates that LLM-based, taxonomy-guided generation can cover a broader and more human-aligned space of reasoning, offering a scalable pathway to bridge human and model explanations. The work also extends the e-SNLI dataset with fine-grained explanation categories and provides methods to assess explanation coverage, with plans to generalize to more variation-aware benchmarks.

Abstract

There is increasing evidence of Human Label Variation (HLV) in Natural Language Inference (NLI), where annotators assign different labels to the same premise-hypothesis pair. However, within-label variation--cases where annotators agree on the same label but provide divergent reasoning--poses an additional and mostly overlooked challenge. Several NLI datasets contain highlighted words in the NLI item as explanations, but the same spans on the NLI item can be highlighted for different reasons, as evidenced by free-text explanations, which offer a window into annotators' reasoning. To systematically understand this problem and gain insight into the rationales behind NLI labels, we introduce LITEX, a linguistically-informed taxonomy for categorizing free-text explanations in English. Using this taxonomy, we annotate a subset of the e-SNLI dataset, validate the taxonomy's reliability, and analyze how it aligns with NLI labels, highlights, and explanations. We further assess the taxonomy's usefulness in explanation generation, demonstrating that conditioning generation on LITEX yields explanations that are linguistically closer to human explanations than those generated using only labels or highlights. Our approach thus not only captures within-label variation but also shows how taxonomy-guided generation for reasoning can bridge the gap between human and model explanations more effectively than existing strategies.

Figures (7)

• Figure 1: Our LiTEx taxonomy reveals within-label variation not captured by highlights: the same highlights can yield different explanations (Example B), and vice versa (Example A).
• Figure 2: Distribution of LiTEx categories on LiTEx-SNLI explanations across NLI labels (n = 3,108).
• Figure 3: Boxplot of explanation similarities grouped by number of LiTEx categories on an NLI item.
• Figure 4: Average number of highlighted words in each premise-hypothesis pair across LiTEx categories.
• Figure 5: Representative t-SNE visualizations of explanation embeddings. The blue convex hull represents the span of human-written explanations, while the gray illustrates the spread of GPT4o-generated explanations.