Data Science Kitchen at GermEval 2021: A Fine Selection of Hand-Picked Features, Delivered Fresh from the Oven

TL;DR

The paper tackles identifying toxic, engaging, and fact-claiming comments in German social media by combining semantic embeddings (from German BERT) with writing-style signals and hand-crafted numerical features within a traditional classifier ensemble framework. It employs three parallel preprocessing streams, a joint 868-dimensional embedding with 28 numerical features, and dimensionality reduction via truncated SVD before training ensembles of Logistic Regression and SVM with hard voting. Calibration metrics reveal some overconfidence, especially for positively labeled instances, but the approach achieves macro-averaged F1-scores of $66.8\%$, $69.9\%$, and $72.5\%$ across the three subtasks, respectively, indicating solid performance in a data-scarce setting. Overall, the work demonstrates that a carefully engineered feature set with conventional models can match advanced architectures in multi-task toxic-content moderation while offering practical benefits in interpretability and calibration.

Abstract

This paper presents the contribution of the Data Science Kitchen at GermEval 2021 shared task on the identification of toxic, engaging, and fact-claiming comments. The task aims at extending the identification of offensive language, by including additional subtasks that identify comments which should be prioritized for fact-checking by moderators and community managers. Our contribution focuses on a feature-engineering approach with a conventional classification backend. We combine semantic and writing style embeddings derived from pre-trained deep neural networks with additional numerical features, specifically designed for this task. Classifier ensembles are used to derive predictions for each subtask via a majority voting scheme. Our best submission achieved macro-averaged F1-scores of 66.8\%,\,69.9\% and 72.5\% for the identification of toxic, engaging, and fact-claiming comments.

Figures (4)

• Figure 1: General architecture of the proposed framework to detect toxic, engaging and fact-claiming comments. Yellow boxes denote non-trainable, computational functions and transformations, green boxes represent pretrained models utilized for feature extraction. Trainable models whose parameters are optimized using the challenge dataset are shown in blue.
• Figure 2: Reliability diagrams of our first submission for all three subtasks (see Section \ref{['subsec:evaluation_metrics']}). The red bars are drawn darker for bins with a higher number of samples.
• Figure 3: Confusion matrices for submission 3.
• Figure 4: Gaussian kernel density estimates for the distributions of the first submission (bandwidth$=0.08$).