Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Argumentative Large Language Models for Explainable and Contestable Claim Verification

Gabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin, Antonio Rago, Francesca Toni

TL;DR

ArgLLMs address the lack of explainability and contestability in large language models by augmenting them with formal argumentative reasoning via quantitative bipolar argumentation frameworks (QBAFs) and the DF-QuAD semantics. The approach decomposes the task of claim verification into argument generation, intrinsic strength attribution, and dialectical strength calculation, producing a final decision and an interpretable reasoning trace. Empirical results show ArgLLMs achieve competitive accuracy with baselines while offering faithful, contestable explanations, and formal proofs establish contestability properties for the underlying semantics. This framework enables robust, explainable decision support suitable for high-stakes domains, with avenues for improvement through retrieval augmentation, ensemble methods, and deeper human evaluations.

Abstract

The profusion of knowledge encoded in large language models (LLMs) and their ability to apply this knowledge zero-shot in a range of settings makes them promising candidates for use in decision-making. However, they are currently limited by their inability to provide outputs which can be faithfully explained and effectively contested to correct mistakes. In this paper, we attempt to reconcile these strengths and weaknesses by introducing \emph{argumentative LLMs (ArgLLMs)}, a method for augmenting LLMs with argumentative reasoning. Concretely, ArgLLMs construct argumentation frameworks, which then serve as the basis for formal reasoning in support of decision-making. The interpretable nature of these argumentation frameworks and formal reasoning means that any decision made by ArgLLMs may be explained and contested. We evaluate ArgLLMs' performance experimentally in comparison with state-of-the-art techniques, in the context of the decision-making task of claim verification. We also define novel properties to characterise contestability and assess ArgLLMs formally in terms of these properties.

Argumentative Large Language Models for Explainable and Contestable Claim Verification

TL;DR

ArgLLMs address the lack of explainability and contestability in large language models by augmenting them with formal argumentative reasoning via quantitative bipolar argumentation frameworks (QBAFs) and the DF-QuAD semantics. The approach decomposes the task of claim verification into argument generation, intrinsic strength attribution, and dialectical strength calculation, producing a final decision and an interpretable reasoning trace. Empirical results show ArgLLMs achieve competitive accuracy with baselines while offering faithful, contestable explanations, and formal proofs establish contestability properties for the underlying semantics. This framework enables robust, explainable decision support suitable for high-stakes domains, with avenues for improvement through retrieval augmentation, ensemble methods, and deeper human evaluations.

Abstract

The profusion of knowledge encoded in large language models (LLMs) and their ability to apply this knowledge zero-shot in a range of settings makes them promising candidates for use in decision-making. However, they are currently limited by their inability to provide outputs which can be faithfully explained and effectively contested to correct mistakes. In this paper, we attempt to reconcile these strengths and weaknesses by introducing \emph{argumentative LLMs (ArgLLMs)}, a method for augmenting LLMs with argumentative reasoning. Concretely, ArgLLMs construct argumentation frameworks, which then serve as the basis for formal reasoning in support of decision-making. The interpretable nature of these argumentation frameworks and formal reasoning means that any decision made by ArgLLMs may be explained and contested. We evaluate ArgLLMs' performance experimentally in comparison with state-of-the-art techniques, in the context of the decision-making task of claim verification. We also define novel properties to characterise contestability and assess ArgLLMs formally in terms of these properties.
Paper Structure (33 sections, 5 theorems, 3 equations, 18 figures, 16 tables)

This paper contains 33 sections, 5 theorems, 3 equations, 18 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Claim Verification
  5. Computational Argumentation
  6. Argumentative LLMs
  7. Argument generation
  8. Intrinsic argument strength attribution
  9. Argumentative strength calculation
  10. Performance Evaluation
  11. Experimental Set-up
  12. Datasets
  13. LLMs
  14. Baselines
  15. Prompt Selection
  16. ...and 18 more sections

Key Result

Proposition 1

DF-QuAD satisfies base score contestability and argument relation contestability.

Figures (18)

  • Figure 1: Comparison of our approach (ArgLLM, here in combination with Mixtral) with existing alternatives. The example claim is adapted from TruthfulQA .
  • Figure 2: Pipeline for ArgLLMs (in comparison with baselines, see §\ref{['sec:main']} and §\ref{['sec:experiments']} for the details).
  • Figure 3: Prompt used for $\Gamma$. {"supporting"/"attacking"} and {"support"/"attack"} are determined by $\theta$.
  • Figure 4: Prompt used for $\mathcal{E}$. {"in favour of"/"against"} and {"supports"/"refutes"} depend on the type of {argument}.
  • Figure 5: An example of contestation, in the ArgLLM with Mixtral, for a claim taken from StrategyClaim . Before contestation, the claim was (incorrectly) classified as False, but after contesting the intrinsic strength of the attacking argument from 0.9 to 0.5 (citing the fallacious reasoning highlighted in red), the correct True classification results.
  • ...and 13 more figures

Theorems & Definitions (13)

  • Example 1
  • Example 2
  • Proposition 1
  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Definition 1
  • Definition 2
  • Proposition 3
  • ...and 3 more