Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting User Stances from Target-Agnostic Information using Large Language Models

Siyuan Brandon Loh, Liang Ze Wong, Prasanta Bhattacharya, Joseph Simons, Wei Gao, Hong Zhang

TL;DR

The findings suggest that LLMs might offer a viable method for determining public stances towards new topics based on historical and target-agnostic data and call for further research to better understand LLMs' strong performance on the stance prediction task and how their effectiveness varies across task contexts.

Abstract

We investigate Large Language Models' (LLMs) ability to predict a user's stance on a target given a collection of his/her target-agnostic social media posts (i.e., user-level stance prediction). While we show early evidence that LLMs are capable of this task, we highlight considerable variability in the performance of the model across (i) the type of stance target, (ii) the prediction strategy and (iii) the number of target-agnostic posts supplied. Post-hoc analyses further hint at the usefulness of target-agnostic posts in providing relevant information to LLMs through the presence of both surface-level (e.g., target-relevant keywords) and user-level features (e.g., encoding users' moral values). Overall, our findings suggest that LLMs might offer a viable method for determining public stances towards new topics based on historical and target-agnostic data. At the same time, we also call for further research to better understand LLMs' strong performance on the stance prediction task and how their effectiveness varies across task contexts.

Predicting User Stances from Target-Agnostic Information using Large Language Models

TL;DR

The findings suggest that LLMs might offer a viable method for determining public stances towards new topics based on historical and target-agnostic data and call for further research to better understand LLMs' strong performance on the stance prediction task and how their effectiveness varies across task contexts.

Abstract

We investigate Large Language Models' (LLMs) ability to predict a user's stance on a target given a collection of his/her target-agnostic social media posts (i.e., user-level stance prediction). While we show early evidence that LLMs are capable of this task, we highlight considerable variability in the performance of the model across (i) the type of stance target, (ii) the prediction strategy and (iii) the number of target-agnostic posts supplied. Post-hoc analyses further hint at the usefulness of target-agnostic posts in providing relevant information to LLMs through the presence of both surface-level (e.g., target-relevant keywords) and user-level features (e.g., encoding users' moral values). Overall, our findings suggest that LLMs might offer a viable method for determining public stances towards new topics based on historical and target-agnostic data. At the same time, we also call for further research to better understand LLMs' strong performance on the stance prediction task and how their effectiveness varies across task contexts.
Paper Structure (18 sections, 2 equations, 4 figures, 3 tables)

This paper contains 18 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. User-Level Stance Dataset
  4. User-Level Stance Prediction Models
  5. LLM-based Models
  6. User-Level Stance Prediction - LLM
  7. User-Level Stance Prediction - LLM (pooled)
  8. Traditional Machine Learning Models
  9. User-Level Stance Detection Benchmark
  10. Experiments and Results
  11. Cross-validation for Non-LLM Methods
  12. Main Results of Performance
  13. ULSP - LLM (pooled) with Thresholding
  14. Relevance of Target-agnostic Tweets
  15. Target-relevant keywords.
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Performance of stance prediction models across three stance targets. Balanced accuracy of ULSP - NonLLM (TFIDF + LogReg), ULSP - LLM (pooled), and ULSP - LLM across "Donald Trump", "Wearing Masks", and "Racial Equality" as a function of tweets per user. Stance detection performance (User-level detection benchmark) is included for comparison.
  • Figure 2: Effect of threshold tuning on the performance of ULSP - LLM (pooled), with ULSP - LLM included for comparison.
  • Figure 3: Biserial correlation coefficients between selected stemmed terms and stance on 3 targets.
  • Figure 4: Biserial correlation coefficients between eMFD scores of target-agnostic tweets and stance on three targets. Cells with an asterisk (*) indicate $p < .01$.