Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explanation Generation for Contradiction Reconciliation with LLMs

Jason Chan, Zhixue Zhao, Robert Gaizauskas

Abstract

Existing NLP work commonly treats contradictions as errors to be resolved by choosing which statements to accept or discard. Yet a key aspect of human reasoning in social interactions and professional domains is the ability to hypothesize explanations that reconcile contradictions. For example, "Cassie hates coffee" and "She buys coffee everyday" may appear contradictory, yet both are compatible if Cassie has the unenviable daily chore of buying coffee for all her coworkers. Despite the growing reasoning capabilities of large language models (LLMs), their ability to hypothesize such reconciliatory explanations remains largely unexplored. To address this gap, we introduce the task of reconciliatory explanation generation, where models must generate explanations that effectively render contradictory statements compatible. We propose a novel method of repurposing existing natural language inference (NLI) datasets, and introduce quality metrics that enable scalable automatic evaluation. Experiments with 18 LLMs show that most models achieve limited success in this task, and that the benefit of extending test-time compute by "thinking" plateaus as model size increases. Our results highlight an under-explored dimension of LLM reasoning and the need to address this limitation in enhancing LLMs' downstream applications such as chatbots and scientific aids.

Explanation Generation for Contradiction Reconciliation with LLMs

Abstract

Existing NLP work commonly treats contradictions as errors to be resolved by choosing which statements to accept or discard. Yet a key aspect of human reasoning in social interactions and professional domains is the ability to hypothesize explanations that reconcile contradictions. For example, "Cassie hates coffee" and "She buys coffee everyday" may appear contradictory, yet both are compatible if Cassie has the unenviable daily chore of buying coffee for all her coworkers. Despite the growing reasoning capabilities of large language models (LLMs), their ability to hypothesize such reconciliatory explanations remains largely unexplored. To address this gap, we introduce the task of reconciliatory explanation generation, where models must generate explanations that effectively render contradictory statements compatible. We propose a novel method of repurposing existing natural language inference (NLI) datasets, and introduce quality metrics that enable scalable automatic evaluation. Experiments with 18 LLMs show that most models achieve limited success in this task, and that the benefit of extending test-time compute by "thinking" plateaus as model size increases. Our results highlight an under-explored dimension of LLM reasoning and the need to address this limitation in enhancing LLMs' downstream applications such as chatbots and scientific aids.
Paper Structure (27 sections, 3 equations, 3 figures, 7 tables)

This paper contains 27 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task formulation
  4. Methodology
  5. REG Dataset Curation
  6. Explanation Generation
  7. Prompting.
  8. Models
  9. Reconciliatory Explanation Evaluation
  10. NLI judge models.
  11. Prompting.
  12. Evaluation metrics
  13. Implementation
  14. Results
  15. Analysis
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Given a premise (P) and hypothesis (H) that a judge model deems contradictory, our novel task requires models to generate a successful explanation (E) such that H is judged as entailed by P combined with E.
  • Figure 2: "Contradiction" label weight ($w_{con.}$) of each instance (i.e. proportion of annotators that labeled it as "Contradiction") in the ChaosNLI-MNLI-C subset (275 instances in total).
  • Figure 3: Explanation success score of each explanation model, as judged by each NLI judge model. For clarity, where Qwen3-[0.6B,1.7B,8B,14B,32B] is run with thinking enabled, "(thinking)" is appended to the model name. A red box encloses a 9 x 9 grid representing the results of the nine NLI judge models when they are themselves assessed as models that generate explanations. The top-left to bottom-right diagonal of this grid represents the explanation success rate of each NLI judge model when it is assessing its own explanations.