Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"The explanation makes sense": An Empirical Study on LLM Performance in News Classification and its Influence on Judgment in Human-AI Collaborative Annotation

Qile Wang, Prerana Khatiwada, Avinash Chouhan, Ashrey Mahesh, Joy Mwaria, Duy Duc Tran, Kenneth E. Barner, Matthew Louis Mauriello

TL;DR

Results show that AI assistance significantly increases confidence, with detailed explanations more persuasive and more likely to alter decisions, and recommendations for AI explanations through thematic analysis are highlighted.

Abstract

The spread of media bias is a significant concern as political discourse shapes beliefs and opinions. Addressing this challenge computationally requires improved methods for interpreting news. While large language models (LLMs) can scale classification tasks, concerns remain about their trustworthiness. To advance human-AI collaboration, we investigate the feasibility of using LLMs to classify U.S. news by political ideology and examine their effect on user decision-making. We first compared GPT models with prompt engineering to state-of-the-art supervised machine learning on a 34k public dataset. We then collected 17k news articles and tested GPT-4 predictions with brief and detailed explanations. In a between-subjects study (N=124), we evaluated how LLM-generated explanations influence human annotation, judgment, and confidence. Results show that AI assistance significantly increases confidence ($p<.001$), with detailed explanations more persuasive and more likely to alter decisions. We highlight recommendations for AI explanations through thematic analysis and provide our dataset for further research.

"The explanation makes sense": An Empirical Study on LLM Performance in News Classification and its Influence on Judgment in Human-AI Collaborative Annotation

TL;DR

Results show that AI assistance significantly increases confidence, with detailed explanations more persuasive and more likely to alter decisions, and recommendations for AI explanations through thematic analysis are highlighted.

Abstract

The spread of media bias is a significant concern as political discourse shapes beliefs and opinions. Addressing this challenge computationally requires improved methods for interpreting news. While large language models (LLMs) can scale classification tasks, concerns remain about their trustworthiness. To advance human-AI collaboration, we investigate the feasibility of using LLMs to classify U.S. news by political ideology and examine their effect on user decision-making. We first compared GPT models with prompt engineering to state-of-the-art supervised machine learning on a 34k public dataset. We then collected 17k news articles and tested GPT-4 predictions with brief and detailed explanations. In a between-subjects study (N=124), we evaluated how LLM-generated explanations influence human annotation, judgment, and confidence. Results show that AI assistance significantly increases confidence (), with detailed explanations more persuasive and more likely to alter decisions. We highlight recommendations for AI explanations through thematic analysis and provide our dataset for further research.
Paper Structure (45 sections, 6 figures, 7 tables)

This paper contains 45 sections, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. News Credibility and Media Bias
  4. Classification Tasks - MTurk and LLMs
  5. Human-AI Collaboration
  6. Social Influence, Collaborative Dynamics, and Human-in-the-Loop AI
  7. AI-Assisted Decision-Making
  8. Bias in AI
  9. Method
  10. Study I - GPT Classification Performance
  11. Dataset
  12. Experiment Design
  13. Study II - Crowdsourcing MTurk Study: Human Feedback
  14. Recruitment and Data Collection
  15. Assessment and Refinement of Response Quality
  16. ...and 30 more sections

Figures (6)

  • Figure 1: Study Design Overview - This diagram illustrates the key components and workflow of our study, including GPT prediction experiments on news articles and human study procedures
  • Figure 2: Survey Flow and User Interface for Classifying Media Bias Tasks. The sample news article shown in the figure is categorized as Center based on the data collection process. Each participant reads three articles from similar stories, one left, one center, and one right, shown in random order.
  • Figure 3: Study Variables: Impact of AI assistance with explanation accuracy and information level on Trust and Decision Making.
  • Figure 4: Distribution of quality measurements for different types of GPT explanation. The differences in each metric are statistically significant ($p < .001$) using t-tests.
  • Figure 5: Confidence levels across all conditions for both initial and final decision with Standard Error (SE). ‘True’ and ‘False’ refer to whether GPT predictions and explanations match the ground truth in the dataset. Differences between initial and final in each group are statistically significant with *** indicates $p < .001$, ** indicates $p < .01$.
  • ...and 1 more figures