Unfair Mistakes on Social Media: How Demographic Characteristics influence Authorship Attribution

TL;DR

This study audits bias in authorship attribution on pseudonymous social media by examining gender, native language, and age through three fairness lenses. It surveys three approaches: (i) sensitivity to demographic composition in training data, (ii) equity of odds in misclassifications under closed-world conditions, and (iii) forced-misclassification to assess demographic-driven error patterns when the true author is absent. Using a novel data-collection pipeline that combines Reddit flair signals with Wayback Machine archives and a reduced writeprints feature set with Logistic Regression, the authors find no overall performance or misclassification disparities across groups in closed-world settings, but reveal demographic-dependent misattribution tendencies in forced, open-world scenarios. These results imply that apparent model fairness in idealized settings can mask biases that emerge under realistic adversarial conditions, informing safer deployment and interpretation of authorship-attribution systems.

Abstract

Authorship attribution techniques are increasingly being used in online contexts such as sock puppet detection, malicious account linking, and cross-platform account linking. Yet, it is unknown whether these models perform equitably across different demographic groups. Bias in such techniques could lead to false accusations, account banning, and privacy violations disproportionately impacting users from certain demographics. In this paper, we systematically audit authorship attribution for bias with respect to gender, native language, and age. We evaluate fairness in 3 ways. First, we evaluate how the proportion of users with a certain demographic characteristic impacts the overall classifier performance. Second, we evaluate if a user's demographic characteristics influence the probability that their texts are misclassified. Our analysis indicates that authorship attribution does not demonstrate bias across demographic groups in the closed-world setting. Third, we evaluate the types of errors that occur when the true author is removed from the suspect set, thereby forcing the classifier to choose an incorrect author. Unlike the first two settings, this analysis demonstrates a tendency to attribute authorship to users who share the same demographic characteristic as the true author. Crucially, these errors do not only include texts that deviate from a user's usual style, but also those that are very close to the author's average. Our results highlight that though a model may appear fair in the closed-world setting for a performant classifier, this does not guarantee fairness when errors are inevitable.

Figures (7)

• Figure 1: Diagram of our data collection methodology. The left column indicates the input and output of each step, and the right column indicates the process used at each step.
• Figure 2: The first-order polynomial predicting a classifier's accuracy based on the native language composition of the suspect set. The points represent the classification accuracy of 10 experiments for each suspect set composition (0 native speakers/16 non-native speakers to 16 native speakers/0 non-native speakers), and the line represents the linear regression result. The slope of the line is almost horizontal. This indicates visually that the performance is not influenced by this type of suspect set composition. Furthermore, the line is well-centred in the point clouds, indicating that it accurately describes the data.
• Figure 3: Boxplots showing the probability distributions of being misclassified depending on your demographic characteristic. This figure recombines multiple experiments; the different shades are labelled by the probability followed by the experimental setup. For the native language experiment, we run two experiments: sNL, where non-native authors share a native language, and rNL, where the non-native authors are randomly chosen.
• Figure 4: Boxplots displaying the probability distribution of intra- and inter-native language assignment for different native languages. $N$ stands for native English speaker, $\neg N$ stands for non-native English speaker. In the experiment setting $randomNL$, non-native authors are randomly chosen, as opposed to the experiment setting $sharedNL$, where non-native authors have a common native language.
• Figure 5: The probability distributions of intra- and inter-gender assignments for different genders.