Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Validating Political Position Predictions of Arguments

Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn

Abstract

Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme \textit{Question Time}. Pointwise evaluation shows moderate human-model agreement (Krippendorff's $α=0.578$), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings ($α=0.86$ for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.

Validating Political Position Predictions of Arguments

Abstract

Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme \textit{Question Time}. Pointwise evaluation shows moderate human-model agreement (Krippendorff's ), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings ( for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.
Paper Structure (42 sections, 5 equations, 6 figures, 6 tables)

This paper contains 42 sections, 5 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs as Evaluators of Language Outputs.
  4. Political Position Prediction with LLMs.
  5. Pairwise Comparison and Preference-Based Evaluation.
  6. Political Argumentation Resources.
  7. Knowledge Base Construction
  8. Data Sources and Preprocessing
  9. LLMs as Judges of Political Position
  10. Ensemble Construction and Aggregation
  11. Ensemble 1 [$E_1$]: All ($n=22$).
  12. Ensemble 2 [$E_2$]: Reasoning Models ($n=5$).
  13. Ensemble 3 [$E_3$]: High-Confidence Models ($n=12$).
  14. Human Annotation and Validation Design
  15. Pointwise Binary Classification of Political Sentiment
  16. ...and 27 more sections

Figures (6)

  • Figure 1: Overview of the methodology used to instantiate a structured argumentative knowledge base containing political positions predictions for all arguments. Green boxes indicate components developed specifically for this study.
  • Figure 2: Nominal Krippendorff's $\alpha_n$ measuring inter-model agreement across data partitions.
  • Figure 3: Distribution of human--model agreement across dataset partitions.
  • Figure 4: Relationship between human--model agreement and model performance metrics.
  • Figure 5: Macro F1 plotted against ordinal human--model agreement for all models and ensembles under $\mathcal{P}$.
  • ...and 1 more figures