Misleading through Inconsistency: A Benchmark for Political Inconsistencies Detection

TL;DR

This paper defines a novel task for detecting inconsistencies in political statements and releasing a dataset of 698 statement pairs (with 237 samples and 334 explanations) drawn from Wahl-O-Mat and Smartvote to benchmark automated inconsistency detection. It introduces a three-category inconsistency taxonomy (Surface, Factual, Indirect) plus Unrelated and Consistent relationships, and demonstrates that large language models can approach or exceed human performance on predicting the general Inconsistent label, though fine-grained subtype detection remains challenging due to labeling variability. The authors present a thorough data-collection and annotation protocol, including crowd-sourced labeling, inter-annotator agreement metrics, and bootstrapped upper-bound estimates to account for subjectivity. They discuss the distinctions between inconsistency detection and related tasks like fact-checking, stance detection, and inconsistency in summarization, and propose practical directions for deploying such systems in the wild, along with limitations and ethical considerations. Overall, the work provides a valuable resource and benchmark for advancing political inconsistency detection and highlights both the potential and the challenges of fine-grained classification in real-world political discourse.

Abstract

Inconsistent political statements represent a form of misinformation. They erode public trust and pose challenges to accountability, when left unnoticed. Detecting inconsistencies automatically could support journalists in asking clarification questions, thereby helping to keep politicians accountable. We propose the Inconsistency detection task and develop a scale of inconsistency types to prompt NLP-research in this direction. To provide a resource for detecting inconsistencies in a political domain, we present a dataset of 698 human-annotated pairs of political statements with explanations of the annotators' reasoning for 237 samples. The statements mainly come from voting assistant platforms such as Wahl-O-Mat in Germany and Smartvote in Switzerland, reflecting real-world political issues. We benchmark Large Language Models (LLMs) on our dataset and show that in general, they are as good as humans at detecting inconsistencies, and might be even better than individual humans at predicting the crowd-annotated ground-truth. However, when it comes to identifying fine-grained inconsistency types, none of the model have reached the upper bound of performance (due to natural labeling variation), thus leaving room for improvement. We make our dataset and code publicly available.

Figures (17)

• Figure 1: Example of inconsistencies from the Green party (left-wing) and the AfD party (far-right), Germany. Underlined are spans that explain the inconsistencies. Original statements can be found in Appendix \ref{['orig_statements_politics']}.
• Figure 2: Schematic description of the data annotation pipelines.
• Figure 3: Annotation switches within the same participant during repeated trials.
• Figure 4: F1-score by model and class.
• Figure 5: Precision and Recall distribution for 3 classes.