Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data

Mykhailo Poliakov, Nadiya Shvai

TL;DR

MisSynth addresses health misinformation by generating context-grounded synthetic fallacy samples via retrieval-augmented generation and fine-tuning with LoRA on MISSCI. The pipeline grounds synthetic reasoning in publication context and demonstrates substantial gains over vanilla baselines across multiple LLMs, with the GPT-5 generated dataset achieving the highest macro F1 of $0.705$ on MISSCI. Notably, small, parameter-efficient models can surpass larger baselines after MisSynth fine-tuning, underscoring the value of high-quality synthetic data for domain-specific reasoning. The work provides a publicly released synthetic dataset and code, offering a practical path to scale specialized fallacy classification when annotated data are scarce.

Abstract

Health-related misinformation is very prevalent and potentially harmful. It is difficult to identify, especially when claims distort or misinterpret scientific findings. We investigate the impact of synthetic data generation and lightweight fine-tuning techniques on the ability of large language models (LLMs) to recognize fallacious arguments using the MISSCI dataset and framework. In this work, we propose MisSynth, a pipeline that applies retrieval-augmented generation (RAG) to produce synthetic fallacy samples, which are then used to fine-tune an LLM model. Our results show substantial accuracy gains with fine-tuned models compared to vanilla baselines. For instance, the LLaMA 3.1 8B fine-tuned model achieved an over 35% F1-score absolute improvement on the MISSCI test split over its vanilla baseline. We demonstrate that introducing synthetic fallacy data to augment limited annotated resources can significantly enhance zero-shot LLM classification performance on real-world scientific misinformation tasks, even with limited computational resources. The code and synthetic dataset are available on https://github.com/mxpoliakov/MisSynth.

MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data

TL;DR

MisSynth addresses health misinformation by generating context-grounded synthetic fallacy samples via retrieval-augmented generation and fine-tuning with LoRA on MISSCI. The pipeline grounds synthetic reasoning in publication context and demonstrates substantial gains over vanilla baselines across multiple LLMs, with the GPT-5 generated dataset achieving the highest macro F1 of on MISSCI. Notably, small, parameter-efficient models can surpass larger baselines after MisSynth fine-tuning, underscoring the value of high-quality synthetic data for domain-specific reasoning. The work provides a publicly released synthetic dataset and code, offering a practical path to scale specialized fallacy classification when annotated data are scarce.

Abstract

Health-related misinformation is very prevalent and potentially harmful. It is difficult to identify, especially when claims distort or misinterpret scientific findings. We investigate the impact of synthetic data generation and lightweight fine-tuning techniques on the ability of large language models (LLMs) to recognize fallacious arguments using the MISSCI dataset and framework. In this work, we propose MisSynth, a pipeline that applies retrieval-augmented generation (RAG) to produce synthetic fallacy samples, which are then used to fine-tune an LLM model. Our results show substantial accuracy gains with fine-tuned models compared to vanilla baselines. For instance, the LLaMA 3.1 8B fine-tuned model achieved an over 35% F1-score absolute improvement on the MISSCI test split over its vanilla baseline. We demonstrate that introducing synthetic fallacy data to augment limited annotated resources can significantly enhance zero-shot LLM classification performance on real-world scientific misinformation tasks, even with limited computational resources. The code and synthetic dataset are available on https://github.com/mxpoliakov/MisSynth.

Paper Structure

This paper contains 28 sections, 12 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fallacy Detection and Scientific Misinformation
  4. Synthetic Data Generation
  5. Fine-Tuning
  6. MISSCI
  7. Other Related Benchmarks
  8. Methodology
  9. RAG for publication context
  10. Synthetic Data
  11. Synthetic fallacious premises.
  12. Synthetic claim–premise pairs.
  13. Prompting and parsing.
  14. Train/validation set for fine-tuning.
  15. Fine-tuning
  16. ...and 13 more sections

Figures (1)

  • Figure 1: Overview of MisSynth synthetic data generation and fine-tuning pipeline. A RAG retrieves an article excerpt ($E_i$) from a source article ($S$) based on a MISSCI claim ($q_i$). This excerpt, along with a dev split sample ($\mathcal{A}$), is used by a Generation LLM to create a synthetic dataset ($\mathcal{D}_{\mathrm{syn}}$). This dataset is then used to fine-tune a model with LoRA, which is finally evaluated on the MISSCI test split.