Are LLMs Good Zero-Shot Fallacy Classifiers?

TL;DR

This paper proposes diverse single-round and multi-round prompting schemes, applying different taskspecific instructions such as extraction, summarization, and Chain-of-Thought reasoning to elicit fallacy-related knowledge and reasoning abilities of Large Language Models (LLMs).

Abstract

Fallacies are defective arguments with faulty reasoning. Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions. However, existing fallacy classifiers are limited by the requirement for sufficient labeled data for training, which hinders their out-of-distribution (OOD) generalization abilities. In this paper, we focus on leveraging Large Language Models (LLMs) for zero-shot fallacy classification. To elicit fallacy-related knowledge and reasoning abilities of LLMs, we propose diverse single-round and multi-round prompting schemes, applying different task-specific instructions such as extraction, summarization, and Chain-of-Thought reasoning. With comprehensive experiments on benchmark datasets, we suggest that LLMs could be potential zero-shot fallacy classifiers. In general, LLMs under single-round prompting schemes have achieved acceptable zero-shot performances compared to the best full-shot baselines and can outperform them in all OOD inference scenarios and some open-domain tasks. Our novel multi-round prompting schemes can effectively bring about more improvements, especially for small LLMs. Our analysis further underlines the future research on zero-shot fallacy classification. Codes and data are available at: https://github.com/panFJCharlotte98/Fallacy_Detection.

Figures (9)

• Figure 1: Examples of fallacies and their types from existing datasets.
• Figure 2: Illustration of single-round and multi-round prompting schemes. (a): Prompt LLMs to classify with or without fallacy type definitions. (b): Prompt LLMs to generate fallacy type definitions and then classify. (c): Prompt LLMs to analyze the input discourse and then classify. (d): Prompt LLMs to warm up (extract, summarize, or infer), analyze the input discourse, and then classify. (e): Prompt LLMs to extract the premises and conclusion, analyze the input discourse, and then classify. (f): Prompt LLMs to reason step by step to classify and then draw an answer.
• Figure 3: Misclassification confusion matrix of common fallacy types given by GPT-4 and Llama3-Chat (8B). Rows are the ground truth fallacy types and columns are predicted fallacy types. Cell values represent the percentages of row fallacy types that are misclassified as column fallacy types.
• Figure 4: Comparison of GPT-4's results under the best performed prompting scheme and the multi-round Premise & Conclusion on Argotario and Reddit.
• Figure 5: Comparison of GPT-4's results under the best performed prompting scheme and the multi-round Premise & Conclusion on Mafalda, Propaganda and Logic.