Addressing Topic Leakage in Cross-Topic Evaluation for Authorship Verification

TL;DR

This work identifies topic leakage as a key threat to reliable cross-topic authorship verification, noting that topic similarity across labeled categories can inflate performance and destabilize model rankings. It proposes Heterogeneity-Informed Topic Sampling (HITS), a principled subsampling framework that constructs smaller but more topic-heterogeneous datasets by iteratively selecting topics with low leakage potential, thereby reducing topic-shortcut reliance. Empirical results on Fanfiction data show that HITS lowers overall scores on some models (indicating higher challenge) but improves ranking stability and reveals topic-dependent biases, especially for topic-fit baselines. To facilitate robust evaluation, the authors introduce RAVEN, a benchmark built with HITS that enables topic shortcut tests and helps identify models that generalize to unseen topics, with reproducible code and data splits for future research.

Abstract

Authorship verification (AV) aims to identify whether a pair of texts has the same author. We address the challenge of evaluating AV models' robustness against topic shifts. The conventional evaluation assumes minimal topic overlap between training and test data. However, we argue that there can still be topic leakage in test data, causing misleading model performance and unstable rankings. To address this, we propose an evaluation method called Heterogeneity-Informed Topic Sampling (HITS), which creates a smaller dataset with a heterogeneously distributed topic set. Our experimental results demonstrate that HITS-sampled datasets yield a more stable ranking of models across random seeds and evaluation splits. Our contributions include: 1. An analysis of causes and effects of topic leakage. 2. A demonstration of the HITS in reducing the effects of topic leakage, and 3. The Robust Authorship Verification bENchmark (RAVEN) that allows topic shortcut test to uncover AV models' reliance on topic-specific features.

Figures (2)

• Figure 1: An illustration of two different scenarios of how texts in various topics are distributed on a topic embedding space. In randomly distributed datasets, some topics are more similar to each other than other topics. When performing a cross-topic train-test split from this data, some topics are leaked into the test data. On the other hand, a topic similarity-controlled dataset removes topics that are similar to each other, reducing the degree of leakage
• Figure 2: A diagram illustrating the pipeline of the HITS method in selecting a topic-heterogeneous subset.