Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition

Tariq Alhindi, Smaranda Muresan, Preslav Nakov

TL;DR

The paper investigates using GPT-3.5 to generate synthetic data to mitigate severe class imbalance in fallacy recognition across five datasets. By fine-tuning an instruction-based T5 model on a mixture of original and synthetic examples under zero-/few-shot conditions, the approach achieves consistent improvements in accuracy and macro-F1 across datasets, with 1-shot prompts often performing best. The study reveals that synthetic data quality and diversity depend on prompt design and that context augmentations provide limited universal gains, emphasizing the need for fallacy-type–aware prompts. It also analyzes how closely synthetic data resemble original data via BLEURT, noting varying similarity across datasets, and releases code and synthetic datasets for public use.

Abstract

Recognizing fallacies is crucial for ensuring the quality and validity of arguments across various domains. However, computational fallacy recognition faces challenges due to the diverse genres, domains, and types of fallacies found in datasets. This leads to a highly multi-class, and even multi-label, setup with substantial class imbalance. In this study, we aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data, thus increasing the representation of the infrequent classes. We experiment with GPT3.5 to generate synthetic examples and we examine the impact of prompt settings for this. Moreover, we explore zero-shot and few-shot scenarios to evaluate the effectiveness of using the generated examples for training smaller models within a unified fallacy recognition framework. Furthermore, we analyze the overlap between the synthetic data and existing fallacy datasets. Finally, we investigate the usefulness of providing supplementary context for detecting fallacy types that need such context, e.g., diversion fallacies. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators. The code and the synthetic datasets are all publicly available.

Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition

TL;DR

The paper investigates using GPT-3.5 to generate synthetic data to mitigate severe class imbalance in fallacy recognition across five datasets. By fine-tuning an instruction-based T5 model on a mixture of original and synthetic examples under zero-/few-shot conditions, the approach achieves consistent improvements in accuracy and macro-F1 across datasets, with 1-shot prompts often performing best. The study reveals that synthetic data quality and diversity depend on prompt design and that context augmentations provide limited universal gains, emphasizing the need for fallacy-type–aware prompts. It also analyzes how closely synthetic data resemble original data via BLEURT, noting varying similarity across datasets, and releases code and synthetic datasets for public use.

Abstract

Recognizing fallacies is crucial for ensuring the quality and validity of arguments across various domains. However, computational fallacy recognition faces challenges due to the diverse genres, domains, and types of fallacies found in datasets. This leads to a highly multi-class, and even multi-label, setup with substantial class imbalance. In this study, we aim to enhance existing models for fallacy recognition by incorporating additional context and by leveraging large language models to generate synthetic data, thus increasing the representation of the infrequent classes. We experiment with GPT3.5 to generate synthetic examples and we examine the impact of prompt settings for this. Moreover, we explore zero-shot and few-shot scenarios to evaluate the effectiveness of using the generated examples for training smaller models within a unified fallacy recognition framework. Furthermore, we analyze the overlap between the synthetic data and existing fallacy datasets. Finally, we investigate the usefulness of providing supplementary context for detecting fallacy types that need such context, e.g., diversion fallacies. Our evaluation results demonstrate consistent improvements across fallacy types, datasets, and generators. The code and the synthetic datasets are all publicly available.
Paper Structure (14 sections, 3 figures, 2 tables)

This paper contains 14 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Fallacy Datasets
  3. Synthetic Data Generation
  4. Experimental Setup
  5. Results
  6. Data Augmentation.
  7. Per-Class Results.
  8. Effect of Additional Context.
  9. Overall Observations.
  10. Original and Synthetic Data Similarity
  11. Related Work
  12. Data Augmentation
  13. Fallacy Recognition
  14. Conclusion and Future Work

Figures (3)

  • Figure 1: Data augmentation and model training pipeline.
  • Figure 2: Example of 1-shot context prompt from the Propaganda dataset of a sentence that has the Irrelevant Authority fallacy and its context.
  • Figure 3: Average BLEURT score (y-axis) between original and synthetic data for each fallacy type in few-shot prompts (x-axis: 1-shot, 2-shot, 5-shot).