Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AustroTox: A Dataset for Target-Based Austrian German Offensive Language Detection

Pia Pachinger, Janis Goldzycher, Anna Maria Planitzer, Wojciech Kusa, Allan Hanbury, Julia Neidhardt

TL;DR

AustroTox introduces a large-scale Austrian German offensive language dataset with token-level spans for vulgarities and targets, addressing the need for interpretable toxicity annotations beyond English. The authors compare fine-tuned German encoders against prompting large language models in zero- and few-shot settings, showing LLMs excel at binary offensiveness while fine-tuned models better capture dialectal vulgarities; they also demonstrate how combining approaches can enhance performance. The dataset construction includes context-aware annotation on DerStandard comments, stratified pre-filtering, and a sensitivity-driven aggregation scheme, with disaggregated annotations published for deeper analysis. Overall, AustroTox provides a valuable resource for targeted moderation research in a German-speaking, dialect-rich context and highlights the complementary strengths of different modeling paradigms for offensive content detection.

Abstract

Model interpretability in toxicity detection greatly profits from token-level annotations. However, currently such annotations are only available in English. We introduce a dataset annotated for offensive language detection sourced from a news forum, notable for its incorporation of the Austrian German dialect, comprising 4,562 user comments. In addition to binary offensiveness classification, we identify spans within each comment constituting vulgar language or representing targets of offensive statements. We evaluate fine-tuned language models as well as large language models in a zero- and few-shot fashion. The results indicate that while fine-tuned models excel in detecting linguistic peculiarities such as vulgar dialect, large language models demonstrate superior performance in detecting offensiveness in AustroTox. We publish the data and code.

AustroTox: A Dataset for Target-Based Austrian German Offensive Language Detection

TL;DR

AustroTox introduces a large-scale Austrian German offensive language dataset with token-level spans for vulgarities and targets, addressing the need for interpretable toxicity annotations beyond English. The authors compare fine-tuned German encoders against prompting large language models in zero- and few-shot settings, showing LLMs excel at binary offensiveness while fine-tuned models better capture dialectal vulgarities; they also demonstrate how combining approaches can enhance performance. The dataset construction includes context-aware annotation on DerStandard comments, stratified pre-filtering, and a sensitivity-driven aggregation scheme, with disaggregated annotations published for deeper analysis. Overall, AustroTox provides a valuable resource for targeted moderation research in a German-speaking, dialect-rich context and highlights the complementary strengths of different modeling paradigms for offensive content detection.

Abstract

Model interpretability in toxicity detection greatly profits from token-level annotations. However, currently such annotations are only available in English. We introduce a dataset annotated for offensive language detection sourced from a news forum, notable for its incorporation of the Austrian German dialect, comprising 4,562 user comments. In addition to binary offensiveness classification, we identify spans within each comment constituting vulgar language or representing targets of offensive statements. We evaluate fine-tuned language models as well as large language models in a zero- and few-shot fashion. The results indicate that while fine-tuned models excel in detecting linguistic peculiarities such as vulgar dialect, large language models demonstrate superior performance in detecting offensiveness in AustroTox. We publish the data and code.
Paper Structure (36 sections, 3 figures, 3 tables)

This paper contains 36 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Creation
  4. Data source
  5. Pre-filtering comments
  6. Annotation campaign
  7. Data aggregation
  8. Inter Annotator Agreement
  9. Cross-validation splits
  10. Experiments
  11. Fine-tuned language models
  12. Prompted LLMs
  13. Evaluation outcomes
  14. Conclusion
  15. Annotators’ Risks
  16. ...and 21 more sections

Figures (3)

  • Figure 1: The posts show the importance of the target of an offensive statement in determining its severity and demonstrate how non-offensive remarks can be vulgar.
  • Figure 2: The LightTag interface
  • Figure 3: The prompt we provide and an example response.