Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media

TL;DR

This work addresses stance detection on social media by leveraging chain-of-thought prompting with ChatGPT (GPT-3.5) in a training-free setting. It compares Direct Question-Answering (DQA) and Step-by-Step QA (StSQA) prompting across SemEval-2016, VAST, and P-Stance datasets, finding that StSQA delivers state-of-the-art results in zero-shot and strong performance in in-domain settings. The study also identifies limitations, including target-specific stance biases, sensitivity to prompt composition, and the need for finer-grained stance definitions. Overall, the results demonstrate the practical potential of VLPLMs with CoT prompts for stance detection and provide guidance on prompt design and bias mitigation for future research.

Abstract

Stance detection predicts attitudes towards targets in texts and has gained attention with the rise of social media. Traditional approaches include conventional machine learning, early deep neural networks, and pre-trained fine-tuning models. However, with the evolution of very large pre-trained language models (VLPLMs) like ChatGPT (GPT-3.5), traditional methods face deployment challenges. The parameter-free Chain-of-Thought (CoT) approach, not requiring backpropagation training, has emerged as a promising alternative. This paper examines CoT's effectiveness in stance detection tasks, demonstrating its superior accuracy and discussing associated challenges.

Figures (3)

• Figure 1: A demonstration of StSQA CoT prompting for stance detection. (The underlined text is the original text).
• Figure 2: The experimental results for LA target with the number of QAPs.
• Figure 3: The difference between word- level and semantic-level prompts.