Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SeLeRoSa: Sentence-Level Romanian Satire Detection Dataset

Răzvan-Alexandru Smădu, Andreea Iuga, Dumitru-Clementin Cercel, Florin Pop

TL;DR

SeLeRoSa addresses the lack of Romanian sentence-level satire datasets by compiling 13,873 labeled sentences from satirical and factual news. The authors evaluate a broad spectrum of baselines, from fine-tuned Romanian pre-trained LMs to open-source LLMs and closed-source GPT models, in zero-shot and fine-tuned settings. They provide two data variants (anonymized and anonymized+pre-processed) and report moderate inter-annotator agreement, highlighting the task's inherent challenges. Fine-tuned models, especially RoGemma 2 9B, achieve the best performance (around 76.9% accuracy and 80.7% F1), while zero-shot performance remains modest though reasoning-based models show promise. The dataset and code are released to stimulate further research in low-resource Romanian NLP for satire detection.

Abstract

Satire, irony, and sarcasm are techniques typically used to express humor and critique, rather than deceive; however, they can occasionally be mistaken for factual reporting, akin to fake news. These techniques can be applied at a more granular level, allowing satirical information to be incorporated into news articles. In this paper, we introduce the first sentence-level dataset for Romanian satire detection for news articles, called SeLeRoSa. The dataset comprises 13,873 manually annotated sentences spanning various domains, including social issues, IT, science, and movies. With the rise and recent progress of large language models (LLMs) in the natural language processing literature, LLMs have demonstrated enhanced capabilities to tackle various tasks in zero-shot settings. We evaluate multiple baseline models based on LLMs in both zero-shot and fine-tuning settings, as well as baseline transformer-based models. Our findings reveal the current limitations of these models in the sentence-level satire detection task, paving the way for new research directions.

SeLeRoSa: Sentence-Level Romanian Satire Detection Dataset

TL;DR

SeLeRoSa addresses the lack of Romanian sentence-level satire datasets by compiling 13,873 labeled sentences from satirical and factual news. The authors evaluate a broad spectrum of baselines, from fine-tuned Romanian pre-trained LMs to open-source LLMs and closed-source GPT models, in zero-shot and fine-tuned settings. They provide two data variants (anonymized and anonymized+pre-processed) and report moderate inter-annotator agreement, highlighting the task's inherent challenges. Fine-tuned models, especially RoGemma 2 9B, achieve the best performance (around 76.9% accuracy and 80.7% F1), while zero-shot performance remains modest though reasoning-based models show promise. The dataset and code are released to stimulate further research in low-resource Romanian NLP for satire detection.

Abstract

Satire, irony, and sarcasm are techniques typically used to express humor and critique, rather than deceive; however, they can occasionally be mistaken for factual reporting, akin to fake news. These techniques can be applied at a more granular level, allowing satirical information to be incorporated into news articles. In this paper, we introduce the first sentence-level dataset for Romanian satire detection for news articles, called SeLeRoSa. The dataset comprises 13,873 manually annotated sentences spanning various domains, including social issues, IT, science, and movies. With the rise and recent progress of large language models (LLMs) in the natural language processing literature, LLMs have demonstrated enhanced capabilities to tackle various tasks in zero-shot settings. We evaluate multiple baseline models based on LLMs in both zero-shot and fine-tuning settings, as well as baseline transformer-based models. Our findings reveal the current limitations of these models in the sentence-level satire detection task, paving the way for new research directions.

Paper Structure

This paper contains 13 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Dataset
  3. Data Collection
  4. Annotation Process
  5. Inter-Annotator Agreement
  6. Data Processing
  7. Dataset Statistics
  8. Experiments and Results
  9. Baselines
  10. Experimental Setup
  11. Results
  12. Discussions
  13. Conclusion

Figures (2)

  • Figure 1: Distribution of sentences across different domains in the dataset
  • Figure 2: Topics distributions identified using BERTopic, on the entire dataset. We grouped topics with less than 2% coverage under the Others category.