Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting contextual information to improve stance detection in informal political discourse with LLMs

Arman Engin Sucu, Yixiang Zhou, Mario A. Nascimento, Tony Mullen

TL;DR

The paper tackles political stance detection in informal online discourse and shows that enriching LLM prompts with structured user-profile summaries derived from historical posts substantially boosts accuracy. Through a three-phase methodology and cross-model evaluation across seven LLMs, it demonstrates accuracy gains of roughly 17.5 to 38.5 percentage points, achieving up to 74% on a real-world politics forum dataset. Key findings highlight that a strategically curated set of 10–20 posts yields near-optimal context, that profile-quality matters more than sheer volume, and that different models contribute complementary strengths in profile generation and classification. These results underscore the practical value of user-level context in improving nuanced political NLP tasks and offer guidance for scalable deployment of contextualized stance detectors.

Abstract

This study investigates the use of Large Language Models (LLMs) for political stance detection in informal online discourse, where language is often sarcastic, ambiguous, and context-dependent. We explore whether providing contextual information, specifically user profile summaries derived from historical posts, can improve classification accuracy. Using a real-world political forum dataset, we generate structured profiles that summarize users' ideological leaning, recurring topics, and linguistic patterns. We evaluate seven state-of-the-art LLMs across baseline and context-enriched setups through a comprehensive cross-model evaluation. Our findings show that contextual prompts significantly boost accuracy, with improvements ranging from +17.5\% to +38.5\%, achieving up to 74\% accuracy that surpasses previous approaches. We also analyze how profile size and post selection strategies affect performance, showing that strategically chosen political content yields better results than larger, randomly selected contexts. These findings underscore the value of incorporating user-level context to enhance LLM performance in nuanced political classification tasks.

Exploiting contextual information to improve stance detection in informal political discourse with LLMs

TL;DR

The paper tackles political stance detection in informal online discourse and shows that enriching LLM prompts with structured user-profile summaries derived from historical posts substantially boosts accuracy. Through a three-phase methodology and cross-model evaluation across seven LLMs, it demonstrates accuracy gains of roughly 17.5 to 38.5 percentage points, achieving up to 74% on a real-world politics forum dataset. Key findings highlight that a strategically curated set of 10–20 posts yields near-optimal context, that profile-quality matters more than sheer volume, and that different models contribute complementary strengths in profile generation and classification. These results underscore the practical value of user-level context in improving nuanced political NLP tasks and offer guidance for scalable deployment of contextualized stance detectors.

Abstract

This study investigates the use of Large Language Models (LLMs) for political stance detection in informal online discourse, where language is often sarcastic, ambiguous, and context-dependent. We explore whether providing contextual information, specifically user profile summaries derived from historical posts, can improve classification accuracy. Using a real-world political forum dataset, we generate structured profiles that summarize users' ideological leaning, recurring topics, and linguistic patterns. We evaluate seven state-of-the-art LLMs across baseline and context-enriched setups through a comprehensive cross-model evaluation. Our findings show that contextual prompts significantly boost accuracy, with improvements ranging from +17.5\% to +38.5\%, achieving up to 74\% accuracy that surpasses previous approaches. We also analyze how profile size and post selection strategies affect performance, showing that strategically chosen political content yields better results than larger, randomly selected contexts. These findings underscore the value of incorporating user-level context to enhance LLM performance in nuanced political classification tasks.
Paper Structure (34 sections, 7 figures)

This paper contains 34 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Political Stance Classification
  4. Contextual LLM Approaches
  5. Personalization and Reasoning in LLMs
  6. Dataset and Preprocessing
  7. Data Source and Characteristics
  8. Data Preprocessing
  9. Methodology and Experimental Design
  10. Experimental Framework Overview
  11. Implementation Approach
  12. Experimental Progression
  13. User Profile Structure
  14. Experiment 1: Contextual Enrichment Impact
  15. Experiment 2: Context Optimization Framework
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Distribution of posts in the data by general class and by a slightly modified version of the writers' own self-descriptions.
  • Figure 2: Classification accuracy comparison with and without user profile summaries.
  • Figure 3: Accuracy by post selection strategy and number of posts used for user profiles.
  • Figure 4: Classification accuracy heatmap by model combination. Profile generation models are shown on the y-axis, while classification models are on the x-axis.
  • Figure : Figure 5: Larger version of Figure 2: Classification accuracy comparison with and without user profile summaries.
  • ...and 2 more figures