Navigating Nuance: In Quest for Political Truth

TL;DR

The paper tackles political bias detection in media by evaluating Llama-3-70B on the MBIB benchmark using a novel prompting framework that embeds subtle Chain-of-Thought reasoning. It compares zero-shot, few-shot, and Chain-of-Thought prompts, showing that CoT prompting achieves the best Macro-F1 across 18 data chunks and can approach, or match, the performance of supervised models like ConvBERT. The study demonstrates the practicality of in-context learning for bias detection, highlighting both the potential and the remaining challenges in generalizability and subtle bias identification. The findings underscore the value of thoughtful prompt design and reasoning scaffolds to improve robustness against misinformation and polarization, with publicly available code and data for reproducibility.

Abstract

This study investigates the several nuanced rationales for countering the rise of political bias. We evaluate the performance of the Llama-3 (70B) language model on the Media Bias Identification Benchmark (MBIB), based on a novel prompting technique that incorporates subtle reasons for identifying political leaning. Our findings underscore the challenges of detecting political bias and highlight the potential of transfer learning methods to enhance future models. Through our framework, we achieve a comparable performance with the supervised and fully fine-tuned ConvBERT model, which is the state-of-the-art model, performing best among other baseline models for the political bias task on MBIB. By demonstrating the effectiveness of our approach, we contribute to the development of more robust tools for mitigating the spread of misinformation and polarization. Our codes and dataset are made publicly available in github.

Figures (2)

• Figure 1: Distribution of text according to length
• Figure 2: Macro-F1 score across 18 chunks for different prompt techniques.