GerAV: Towards New Heights in German Authorship Verification using Fine-Tuned LLMs on a New Benchmark

TL;DR

This work introduces GerAV, a large German-authorship verification benchmark with over 600k labeled text pairs from Reddit and Twitter across in-domain, cross-domain, and profile-based settings. It evaluates a broad spectrum of baselines, zero-shot LLMs, and fine-tuned LLMs (notably Gemma-3-12b) using LoRA, showing that tuned open-source LLMs outperform traditional AV methods and even GPT-5 in some conditions. Key findings include a clear benefit of mixing training sources for cross-domain generalization and a strong impact of input length on performance, with profile concatenation yielding higher accuracy. The work establishes GerAV as a robust, versatile benchmark for German AV and cross-domain AV research, and identifies directions for interpretability and cross-language extension.

Abstract

Authorship verification (AV) is the task of determining whether two texts were written by the same author and has been studied extensively, predominantly for English data. In contrast, large-scale benchmarks and systematic evaluations for other languages remain scarce. We address this gap by introducing GerAV, a comprehensive benchmark for German AV comprising over 600k labeled text pairs. GerAV is built from Twitter and Reddit data, with the Reddit part further divided into in-domain and cross-domain message-based subsets, as well as a profile-based subset. This design enables controlled analysis of the effects of data source, topical domain, and text length. Using the provided training splits, we conduct a systematic evaluation of strong baselines and state-of-the-art models and find that our best approach, a fine-tuned large language model, outperforms recent baselines by up to 0.09 absolute F1 score and surpasses GPT-5 in a zero-shot setting by 0.08. We further observe a trade-off between specialization and generalization: models trained on specific data types perform best under matching conditions but generalize less well across data regimes, a limitation that can be mitigated by combining training sources. Overall, GerAV provides a challenging and versatile benchmark for advancing research on German and cross-domain AV.

Figures (8)

• Figure 1: Overview of our approach. The figure presents the GerAV benchmark and its structure, illustrating how the datasets are used as input for AV models during training and testing. Based on a decision threshold, the models predict whether a given pair of texts was written by the same or by different authors.
• Figure 2: Log frequencies of post word lengths for the Reddit (blue) and Twitter (orange) dataset before preprocessing. The long tail of the Reddit distribution is truncated at 0.05% for visualization purposes; the true maximum post length is 4,321 words.
• Figure 3: F1-Score for the baselines and our GerAV models. The y-axis shows the model names and the x-axis shows the evaluated test set. The brackets behind the model indicate its training set: Mix Reddit Twitter (mix), Reddit Cross Domain (rcd), Reddit In Domain (rid), Reddit Profile Based (rpb) and Twitter (tw). The subscript with $t$ indicates the validation set that was used to tune the decision threshold for the baselines. The highest value in every column is written in bold.
• Figure 4: F1-Score per message length in words (up to 2500). $\rho$ is the Spearman correlation between message length and F1 scores. We display the results with bucket sizes increasing on a log-scale, to account for less examples existing for longer messages.
• Figure 5: Accuracy and F1-scores of GPT-5 and GerAV: gemma-3-12 (mix). They are evaluated on a stratified 480 sample subset of the mixed test set.